If you’re reading this post, it’s no thanks to Dreamhost.Â The past few days I have noticed my web traffic dropping off substantially.Â I pay $15/mo. plus an annual fee for their Private Server offering in order to host this and a few other blogs.Â I discovered a great many problems occurring when I checked the server, such as browsers getting a “500 Internal Server Error” on attempting to load documents on all of my WordPress sites.
Rifling through the error.log, I found these events clustering and generally taking the form:
[Mon Jan 19 23:15:08 2009] [error] [client 220.127.116.11] (12)Cannot allocate memory: couldn't create child process: 12: php5.cgi [Mon Jan 19 23:15:08 2009] [error] [client 18.104.22.168] (12)Cannot allocate memory: couldn't spawn child process: /dh/cgi-system/php5.cgi [Mon Jan 19 23:15:08 2009] [error] [client 22.214.171.124] (12)Cannot allocate memory: couldn't create child process: 12: php5.cgi [Mon Jan 19 23:15:08 2009] [error] [client 126.96.36.199] (12)Cannot allocate memory: couldn't spawn child process: /dh/cgi-system/php5.cgi [Mon Jan 19 23:15:21 2009] [error] [client 188.8.131.52] Premature end of script headers: php5.cgi [Mon Jan 19 23:15:21 2009] [error] [client 184.108.40.206] Premature end of script headers: php5.cgi [Mon Jan 19 23:15:48 2009] [error] [client 220.127.116.11] Premature end of script headers: php5.cgi [Mon Jan 19 23:15:48 2009] [error] [client 18.104.22.168] Premature end of script headers: php5.cgi
… thus leading to the 500 errors on the client.
I dutifully emailed support.Â I noticed this announcement from around the same time I started debugging, however it didn’t mention my specific Private Server.Â I got a response from support from Liz advising me that:
I usually see that error when there is a WordPress plugin not playing nicely with PHP. Since you should be able to get into the admin panel for the WP install (http://xxxxxxxxxx/wp-login.php, where “xxxxxxxxxx” is the domain name of each domain running WP), I would recommend going in and turning off your plugins, and then turning them back on one by one until you hit the one that breaks your site, and then seeing if it needs to be updated.
This seemed plausible since I had recently made extensive renovations to one of my sites.Â I spent hours ripping apart and tearing down changes I had made, baselining the offending blog.Â This yielded no improvement.Â I then downgraded PHP, turned off memory-intensive cacheing operations and similar gadgets, and executed a couple of reboots.Â I sent several replies to the Support email (which, stupidly, spawns new tickets rather than amending old ones) and ever since have not received a response.
Similarly, updates to the Dreamhost Status blog have ceased.Â It’s as though no one works there anymore.
The problem is always the same.Â My PS is good for a half hour or so after a reboot and then starts throwing errors like above, ultimately failing completely.Â Then I did some googling and I realized that I am not alone.
Specifically, the comments @ Dreamhost Status and Twitter are revealing that this is common and that there is presently a major outage.Â Disastrously, Dreamhost are not talking to anyone about it.Â No updates to theirÂ status blog, no email responses from Customer Support.Â Just a wild goose chase which wasted hours of my evening and morning debugging something that is essentially their issue.
This also seems to have revealed that Dreamhost has a standard practise of nuking processes, via script, at random when their virtual hosting architecture becomes stressed.Â That would certainly explain why my web site, which served only 28 people today, is maxing out and failing to allocate memory to new processes at an already more-than-adequate 150MB of memory service package with VPS.
I also discovered that Dreamhost vastly over-reports memory usage via its VPS control panel, which would encourage the unwise to make radical upgrades to their service accounts in order to try to address problems like mine, wasting their money:
… this does not reconcile with the Unix top command which reveals nothing extraordinary happening on my virtual private server:
If you see similar stats, don’t upgrade your VPS account — it’s a scam.Â Dreamhost is trying to bully you into upgrading your service.
Anyway, as of this posting I have rebooted my VPS 7 times in the past 24 hours, and had to do it again just in order to submit this entry.Â Not that any of you will be able to read it until I am finished moving to a new host, anyway..
So here are the five deadly sins I think we’ve caught Dreamhost in today:
- Misreporting a service-wide fault as a localized (and neutralized) problem
- Misleading customers via support instructing them to debug when clearly the situation is 1)
- Misleading customers via so-called “management” tools that report extremely inflated resource consumption statistics in order to upsell them
- Allowing such problems as 1) to carry on for 24+ hours without a useful update or resolution advice
- Sucking.Â hard.
So… bye guys.Â Enjoy the chargebacks!
UPDATE:Â see the exciting sequel to this story — Dreamhost Customer Service FAIL.