Rifling through the error.log, I found these events clustering and generally taking the form:

[Mon Jan 19 23:15:08 2009] [error] [client 24.6.13.227]
  (12)Cannot allocate memory: couldn't create child process: 12: php5.cgi
[Mon Jan 19 23:15:08 2009] [error] [client 24.6.13.227]
  (12)Cannot allocate memory: couldn't spawn child process: /dh/cgi-system/php5.cgi
[Mon Jan 19 23:15:08 2009] [error] [client 24.6.13.227]
  (12)Cannot allocate memory: couldn't create child process: 12: php5.cgi
[Mon Jan 19 23:15:08 2009] [error] [client 24.6.13.227]
  (12)Cannot allocate memory: couldn't spawn child process: /dh/cgi-system/php5.cgi
[Mon Jan 19 23:15:21 2009] [error] [client 24.6.13.227]
  Premature end of script headers: php5.cgi
[Mon Jan 19 23:15:21 2009] [error] [client 24.6.13.227]
  Premature end of script headers: php5.cgi
[Mon Jan 19 23:15:48 2009] [error] [client 24.6.13.227]
  Premature end of script headers: php5.cgi
[Mon Jan 19 23:15:48 2009] [error] [client 24.6.13.227]
  Premature end of script headers: php5.cgi

… thus leading to the 500 errors on the client.

I dutifully emailed support.  I noticed this announcement from around the same time I started debugging, however it didn’t mention my specific Private Server.  I got a response from support from Liz advising me that:

I usually see that error when there is a WordPress plugin not playing nicely with PHP. Since you should be able to get into the admin panel for the WP install (http://xxxxxxxxxx/wp-login.php, where “xxxxxxxxxx” is the domain name of each domain running WP), I would recommend going in and turning off your plugins, and then turning them back on one by one until you hit the one that breaks your site, and then seeing if it needs to be updated.

This seemed plausible since I had recently made extensive renovations to one of my sites.  I spent hours ripping apart and tearing down changes I had made, baselining the offending blog.  This yielded no improvement.  I then downgraded PHP, turned off memory-intensive cacheing operations and similar gadgets, and executed a couple of reboots.  I sent several replies to the Support email (which, stupidly, spawns new tickets rather than amending old ones) and ever since have not received a response.

Similarly, updates to the Dreamhost Status blog have ceased.  It’s as though no one works there anymore.

The problem is always the same.  My PS is good for a half hour or so after a reboot and then starts throwing errors like above, ultimately failing completely.  Then I did some googling and I realized that I am not alone.

Specifically, the comments @ Dreamhost Status and Twitter are revealing that this is common and that there is presently a major outage.  Disastrously, Dreamhost are not talking to anyone about it.  No updates to their  status blog, no email responses from Customer Support.  Just a wild goose chase which wasted hours of my evening and morning debugging something that is essentially their issue.

This also seems to have revealed that Dreamhost has a standard practise of nuking processes, via script, at random when their virtual hosting architecture becomes stressed.  That would certainly explain why my web site, which served only 28 people today, is maxing out and failing to allocate memory to new processes at an already more-than-adequate 150MB of memory service package with VPS.

I also discovered that Dreamhost vastly over-reports memory usage via its VPS control panel, which would encourage the unwise to make radical upgrades to their service accounts in order to try to address problems like mine, wasting their money:

… this does not reconcile with the Unix top command which reveals nothing extraordinary happening on my virtual private server:

If you see similar stats, don’t upgrade your VPS account — it’s a scam.  Dreamhost is trying to bully you into upgrading your service.

Anyway, as of this posting I have rebooted my VPS 7 times in the past 24 hours, and had to do it again just in order to submit this entry.  Not that any of you will be able to read it until I am finished moving to a new host, anyway..

So here are the five deadly sins I think we’ve caught Dreamhost in today:

1. Misreporting a service-wide fault as a localized (and neutralized) problem
2. Misleading customers via support instructing them to debug when clearly the situation is 1)
3. Misleading customers via so-called “management” tools that report extremely inflated resource consumption statistics in order to upsell them
4. Allowing such problems as 1) to carry on for 24+ hours without a useful update or resolution advice
5. Sucking.  hard.

So… bye guys.  Enjoy the chargebacks!

UPDATE:  see the exciting sequel to this story — Dreamhost Customer Service FAIL.

Consumers strike back, sue RIAA By Liane Cassavoy, PC World.com September 12, 2003 9:35 am ET

After taking its antipiracy campaign to court, the music industry is finding itself on the receiving end of a lawsuit that challenges its purported amnesty program as a fraudulent business practice.

The Recording Industry Association of America Inc. announced its Clean Slate program Monday, when it filed suit against 261 people for copyright infringement as a result of excessive use of peer-to-peer services. The Clean Slate program purports to offer amnesty to repentant file-swappers who promise to stop using peer-to-peer services to illegally download copyrighted works and to destroy any copies of downloaded audio files.

To qualify for the amnesty program, applicants must fill out a sworn affidavit that requires a full name, address, telephone number and e-mail address, have it notarized, and send it to the RIAA. In turn, the RIAA agrees not to “support or assist in any copyright infringement suits based on past conduct,” according to the organization.

But the offer is neither clean nor a sweep, says Ira Rothken, the Marin County attorney who filed the consumer lawsuit Tuesday in California Superior Court.

Not the RIAA’s Call?

The RIAA claims the amnesty program “would provide people with a clean slate, but after a further reading of the legal documents, it became apparent that this Clean Slate program didn’t provide any such thing,” Rothken says.

“The legal document provides no release of claims, no promise not to sue you. It offers no promise to actually clean the slate by destroying the data that these people provide,” he adds. “All it says is that the RIAA simply will not cooperate in any lawsuit brought against you. That on its face is a deceptive business practice.”

And the offer is deceptive because the RIAA does not own the copyrights in question, Rothken says. The music labels — RIAA members — are the plaintiffs, he says. But because the RIAA is leading the charge, people think the RIAA has the power to promise not to sue them, when it doesn’t, Rothken says.

“Any of the RIAA’s members could file suit against these individuals who have participated in the Clean Slate program, and subpoena the information they need from the RIAA about this person’s guilt,” Rothken says. “So, in the end, the person who supplies all this information to the Clean Slate program will have a dirtier slate than they would have if they never participated.”

RIAA Stands Firm

The RIAA disputes this interpretation, saying the affidavit form, which is available on MusicUnited, is not deceptive.

“Read the form, it’s pretty clear what’s being offered and who’s offering it,” says Jonathan Lamy, an RIAA spokesperson.

“Apparently no good deed goes unpunished,” Lamy adds of the criticism. “It’s also unfortunate and ironic that some lawyer would try to prevent others from getting the assurance that they want, that they won’t be sued.”

Rothken’s law firm is not the first organization to warn people about potential danger in the RIAA’s amnesty program. Last week, before the RIAA formally announced its plan, the digital rights group Electronic Frontier Foundation warned users against accepting the RIAA’s amnesty offer.

“Stepping into the spotlight to admit your guilt is probably not a sensible course for most people sharing music files online, especially since the RIAA doesn’t control many potential sources of lawsuits,” Wendy Seltzer, EFF staff attorney, said in a statement last week.

Murky Music

Cary Sherman, the RIAA’s president, addressed these concerns Monday when announcing the copyright crackdown and amnesty deal. He rebutted suggestions that participating in the Clean Slate program could prove costly.

“We have pledged to keep this information solely for our use, for our records of people who should not be sued,” Sherman said. He said the RIAA would not release the data to copyright holders who might intend to sue.

Unconvinced, Rothken went a step further than simply warning consumers about the program’s potential pitfalls and filed a complaint under the California Business and Professions Code. His lawsuit asks the court to end the RIAA’s program as “unlawful, unfair, and deceptive.” The RIAA’s “guarantee not to sue file sharers” is designed to mislead the public into incriminating themselves by giving the RIAA “admissions of wrong-doing.”

The RIAA’s intentions remain unclear, says Deborah Peckham, a partner in the Patent and Intellectual Property Practice Group at Testa, Hurwitz & Thibeault in Boston.

“It’s really too soon to say whether these people are putting themselves at risk by participating in this program, but the allegations made in (Rothken’s lawsuit) are not without some merit,” Peckham says. “In essence, what the RIAA is doing apparently is collecting these affidavits and storing the information somewhere. The concern is that that information is going to be spread among the music companies, and there’s nothing in the agreement that would bar those companies from suing the users.”

Peckham has reviewed some of the suits filed Monday, and confirms that the RIAA is not named as plaintiff in any she has seen.

California’s Standards

Rothken’s complaint cites California law that says fraud may exist not only if a consumer is injured by the business practice in question, but if there is the potential for injury.

“The only standard is that the business practice in question is likely to deceive reasonable members of the general public,” Rothken says.

This law also allows one citizen to be named as a plaintiff to represent the general population. In this lawsuit, that citizen is Eric Parke, a former paralegal who has not used peer-to-peer networks to download music illegally, according to Rothken.

“In California, our law allows people who are unaffected by the business practice in question, people who would not have traditional standing to sue, to serve as the plaintiff,” he says. One reason for this, Rothken says, is that people who stand up against large organizations could be retaliated against, and may be unwilling to come forward.

The lawsuit now goes to the California Superior Court, but Rothken cannot guess when the case might be heard. He does expect the court will insist that the RIAA make good any amnesty offer.

“The court will likely tell the RIAA that if they’re going to promise amnesty and a clean slate, then you have to do something that delivers on that promise; for example, you have to offer a release of all claims. Or, if you can’t do that, you have to stop the promise, don’t call it amnesty,” Rothken says. “It’s likely the RIAA will have to admit that they don’t have the authority to release all claims, because they don’t have the power to stop these lawsuits, because they don’t own the copyrights.”

Apple originally licensed the name from Apple Records, agreeing not to include sound on their computers and to stay away from the music industry. Look in the system folder on your Mac, you’ll see (and hear) a sound effect called “Sosumi”. This has been included on every Apple computer since sound was first introduced. Wanna know where that name comes from? So, sue me.

-Ian.

—– http://maccentral.macworld.com/news/2003/09/11/applemusic/ index.php?redirect63293387000

By Jim Dalrymple jdalrymple [at] maccentral [dot] com September 11, 2003 6:00 pm ET

Apple Computer Inc. is being sued by Apple Corps. The parent company for music legends, The Beatles, has begun legal proceedings against Apple Computer, citing breach of contract for the suit, according to Fox News .

Apparently when Apple Computer first started, The Beatles sued them for the use of the corporate name. In addition to a hefty cash settlement, Apple agreed to only use the corporate name for computer products and not enter the music markeplace.

Years later, The Beatles sued and won another lawsuit when Apple shipped computers that allowed music to be played through attachable speakers. That lawsuit charged breach of a trademark agreement since Apple had agreed to steer clear of the music business. Fox News estimates Apple has paid US$50 million in the lost suits so far.

The latest round of legal proceedings surround Apple’s popular MP3 player, the iPod and the iTunes Music Store, which just sold its 10 millionth song online.

“When it first happened with the iPod, we said, “What could they be thinking?” said a Beatles legal insider, who agreed that posters announcing the iPod from “AppleMusic” were among the most egregious violations. “They knew we had the agreement, and that we’d won a lot of money from them already.”

An Apple representative was not immediately available for comment.