Well... its been a hectic couple hours for both myself and the CHAOS website. Sometime early last night an automated tool started running against our website doing what is called a web spider, where it loads every page and then every link on that page over and over sending various garbage data and other things. People use these tools to go over every website on the Internet looking for vulnerabilities and this is just part of hosting a site. This wasn't the first time the CHAOS website was "under the gun" from a spider, it happens multiple times every week. In fact, I do IT security professionally as my job, so I have even purposefully put the website through these tests to try and make sure this didn't happen.
However, this particular scan was a much higher volume that any we've undergone previously. At its peak, it was sending over 10,000 requests to load pages per second for several hours. You can see in the graph below (times are GMT) of the CPU usage of our web server, on the left side this is the last part of the scan. As I said before, this happens multiple times a week, but the CPU never goes above 30-40%.
The web-server handled things just fine, but due to the high volume the backend database had a difficult time handling all of the attempts at reading and inserting data. At some point the database got corrupted and the second CPU peak was where the database was trying for several hours to recover, but it was unable. At around 7:30 tonight the database simply crashed and all of our data was corrupted.
Thankfully, we keep backups for this exact reason. I have restored the website and database from a backup taken at approximately 1PM today. I attempted to restore from a later backup, but the database was unable to import the data as it was already too corrupted.
At this point there are a few important points:
- The site does not appear to be compromised, but even if it was WE DON'T STORE ANY SENSITIVE DATA ABOUT YOU except your CHAOS password and street address. That's right, we don't keep your paypal info, credit card info, etc. That is why we use PayPal and Stripe. And your address... well that's in the Address Book and is on public record. Even so, we highly value your privacy and security and have taken every precaution to protect even this information.
- It's never a good idea to share passwords between websites and accounts. Take a look at a tool like KeePass (http://keepass.info) to generate and manage your passwords.
- There were a few posts, private messages, and other items that were lost. I'm sorry, but they're gone and there's nothing I can do...
If you have any other questions please let me know, I'll be at the meeting tomorrow. For now, I'm going to go get a beer as I'm exhausted trying to get things back up.
