Over the course of a day, my server rebooted httpd twice. That’s not a common thing for me, as after days and hours of work, I managed to finegal my server into a semblance of stability so it could handle massive loads. At first I thought it was due to my traffic spike of an increase of about 1500% (no I did not screw up a decimal place there, my traffic went from a couple hundred to nearly 4000 in one day). Then I thought that couldn’t be right, because my traffic had actually mellowed out by the time I was crashing.
So I went to my emails and pulled up the Apache Status log and noticed that 70% of the calls to my site were GETs to pages like this:
And thanks to that massive spike in traffic, my server was slowing down to the point that HTTP was becoming unresponsive and it had to reboot itself. In short, the TimThumb exploit was causing my server to behave like it was under a Denial of Service Attack, even though I don’t use TimThumb! My server was able to handle this, but if I’d been back on my old Shared Server, I’d probably have not gotten a text from the server at 11pm saying “Man, we had to reboot, but it’s okay now. Can I have a beer?”, but instead woken up to ‘Dude, where’s my website!?’ And this is with having a fantastic web host who cares and takes the time to help me out.
Normally this is where I’d tell you what to do if you’ve been infected via the TimThumb exploit, but Cleaning Up the TimThumb Hack covered it pretty well. Just remember this, if you have been infected, you must reset all your passwords. This is true of any and all hacks. As soon as someone has access to meddle with files on your server, you could be hurt worse than you know. At the very least, you need to read the post “Technical details and scripts of the WordPress Timthumb.php hack” by the guy who ‘fixed’ TimThumb.
What I wanted to do here was sort out how to block people who were looking for timthumb.php files (I can’t block thumb.php as I use that filename elsewhere). Reading up on Perishable Press’s Stupid .htaccess Tricks it’s clear we can do this:
# prevent viewing of a specific file
order allow,deny deny from all
That should simply block access. An .htaccess block is a pretty simple way to reduce your server load, because people are getting punted before they get very far into things. Still, it’s something I have to install on each account on my server. Right now they’re just hammering ipstenu.org, and this is not the only domain on my server. This is, by the way, the same problem with using a plugin like WordPress Firewall. It’s a fantastic idea, if all you have is one account on a server. Protect yourself.
I don’t. I run a VPS, and I have four domains here which I should be protecting. It’s easy enough to make that change on all four, plus two other external servers, but is that the best use of my time? I say no. I think I should automate as much of this as I can. What I really want is to say ‘If you’re looking for timthumb.php, then the odds are you’re looking for security vulnerabilities, and you need to just die in a fire.’ Or at least firewall block them. Seeing as I already have CSF on my server, it was logical for me to start there.
Blocking an IP is easy, and I can do it via command line or GUI. Auto-detecting a URL though, is more complicated. Part of me thinks that, much like I can auto-block someone who tries to log in with any ID 10 times in a row, I should be able to add in a file pattern somewhere. Turns out you can’t, at least not the way I wanted. Instead, you have to do it differently.
TimThumb’s exploit scanner isn’t actually a DDoS Attack, but it acts like one. A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make your site unavailable. Basically they hit your site with so much traffic that it’s incapable of its job: serving up webpages to happy readers! That’s why I call this a pseudo (D)DoS attack. The goal of the scanner is to see if you’re using Timthumb and, if so, to be evil. It’s not really distributed (i.e. multiple computers at once), though because of the number of people running the exploit scanner, it can seem that way. The side effect is that your site is hammered and can’t do what it wants. Which leads us to Connection Tracking.
CSF has a tool called ‘Connection Tracking’ which lets you limit how many times a specific IP can hit your site at once before they get tossed to a shit-list. I set this to 300, and told it to only scan ports 80 and 443 (because I need to have unlimited FTP, and sometimes I log in from different IPs – yes, my home IP is on the whitelist).
Connection Tracking. This option enables tracking of all connections from IP addresses to the server. If the total number of connections is greater than this value then the offending IP address is blocked. This can be used to help# prevent some types of DOS attack.
Care should be taken with this option. It’s entirely possible that you will see false-positives. Some protocols can be connection hungry, e.g. FTP, IMAPD and HTTP so it could be quite easy to trigger, especially with a lot of closed connections in TIME_WAIT. However, for a server that is prone to DOS attacks this may be very useful. A reasonable setting for this option might be around 300.
Setting this up is a little less obvious for the new person. Go to WHM > Plugins > ConfigServer Security & Firewall and look for “Firewall Configuration”
Click on the button, look for
CT_LIMIT and change it to 300.
Scroll down, click ‘Change’ and then restart CSF.
Now, you could put this as low as 100, or as high as you want, but I did some reading and 300 seems like something not too likely to trip innocent people up, but just enough to catch the bad guys. I may way to lower this to 200 or so, but I know that a lot of people come to one part of my server for the image gallery, and they tend to open lots of pages at once. I don’t want to hurt them. The other thing to keep in mind is how short is the block time. The IP block is set for 30 minutes, which isn’t much at all, but it could be just enough to make the transient DDoSers go away. ETA: As of February 2012, I’ve actually lowered this to 50, and it’s made a world of difference! My day to day CPU is a little higher, but the number of spikes that caused outages has dropped.
I’m not doing this to stop the people who want to bring my server to its knees. I’m doing it to stop the people who are ‘scanning’ for exploits. A true DDoS is hard to block because as soon as I block it, I have to defend against it again and again. CSF would be like a sump pump in your flooded basement, always running until it burns out the motor. It comes from too many sources, and for the little guy (i.e. me), I may just have to shut things down for an hour and wait it out. But these scanners, well, I can block them with this trick, and not hurt my server doing so!