I’ve mentioned before that this particular site exists because once in a blue moon, my regular site runs into technical difficulties, sometimes within my capacity to fix, sometimes not so much.
Yesterday I had no such problems, but the host’s own domain was down for several hours, and this is the explanation proffered:
We’ve got a fairly liberal free-speech policy here which we’re quite proud of. Speech that is protected by the United States Constitution’s First Amendment is protected by DreamHost. While we don’t always agree with the content of the sites we host, we do support their right to host it in America!
Yesterday was Draw Mohammad Day.
This did not sit well with roughly 21% of the world’s population.
We happened to be hosting drawmuhammadday.com, a site that encouraged people to draw images of Mohammed. That’s kind of a no-no in the Muslim world.
Incidentally, did you know there’s like a million different ways to spell Mohammed?
Anyway, “did not sit well” translated into this:
Some people weren’t too keen on the idea of the Draw Mohammad Day website and suddenly we were the target of the largest Distributed Denial of Service attack (DDoS) we’ve ever seen. drawmuhammadday.com was the first to fall. It was the main target and it didn’t take long … based on our stats it looked like almost the entire country of Pakistan was attacking us! Well not really. But nobody in Pakistan could reach YouTube, Facebook, or Twitter yesterday, so what else were they gonna do?
They did not let up once the site was down. At one point dreamhost.com (the site itself) was handling around 20,000 requests per second. To put that number in perspective, when our customers’ sites have traffic surges a busy day might see that number get up to ten or even twenty.
Extraordinary circumstances call for extraordinary measures:
To restore services we had to take the site down altogether while we moved it to newer, stronger hardware, beyond the reach of our load balancers. We tuned the Linux kernel on this new machine aggressively to use less memory for TCP connections. We also abandoned Apache, favoring a specialized nGinx installation.
Downtime at my site? Zero. Zilch. Nada. Bupkis.
Thought you might like to know.