All sorts of fun

I’ve mentioned before that this particular site exists because once in a blue moon, my regular site runs into technical difficulties, sometimes within my capacity to fix, sometimes not so much.

Yesterday I had no such problems, but the host’s own domain was down for several hours, and this is the explanation proffered:

We’ve got a fairly liberal free-speech policy here which we’re quite proud of. Speech that is protected by the United States Constitution’s First Amendment is protected by DreamHost. While we don’t always agree with the content of the sites we host, we do support their right to host it in America!

Yesterday was Draw Mohammad Day.

This did not sit well with roughly 21% of the world’s population.

We happened to be hosting, a site that encouraged people to draw images of Mohammed. That’s kind of a no-no in the Muslim world.

Incidentally, did you know there’s like a million different ways to spell Mohammed?

Anyway, “did not sit well” translated into this:

Some people weren’t too keen on the idea of the Draw Mohammad Day website and suddenly we were the target of the largest Distributed Denial of Service attack (DDoS) we’ve ever seen. was the first to fall. It was the main target and it didn’t take long … based on our stats it looked like almost the entire country of Pakistan was attacking us! Well not really. But nobody in Pakistan could reach YouTube, Facebook, or Twitter yesterday, so what else were they gonna do?

They did not let up once the site was down. At one point (the site itself) was handling around 20,000 requests per second. To put that number in perspective, when our customers’ sites have traffic surges a busy day might see that number get up to ten or even twenty.

Extraordinary circumstances call for extraordinary measures:

To restore services we had to take the site down altogether while we moved it to newer, stronger hardware, beyond the reach of our load balancers. We tuned the Linux kernel on this new machine aggressively to use less memory for TCP connections. We also abandoned Apache, favoring a specialized nGinx installation.

Downtime at my site? Zero. Zilch. Nada. Bupkis.

Thought you might like to know.


One thought on “All sorts of fun

  1. […] So far as I can tell, we had no downtime on this site yesterday. Explanation here. […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s