Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: firewall

  • March into CloudFlare

    March into CloudFlare

    “Eating your own dogfood” is a colloquialism that describes a company using its own products or services for its internal operations. Microsoft supposedly invented it in the 1980s.

    DreamHost is partners with CloudFlare. I’ve tried time and again but I get hung up when we start talking about the delay in caching and the proxying of caches. Still, I know me. I can do things better and provide better support if I use a thing. I’m so good with PageSpeed and wp-cli now because I use them regularly.

    So it was time to knuckle down and use it for more than just a week. I was committing to a month on CloudFlare!

    Pick One Domain

    I decided to only do this for one domain. My busiest, and not a multisite, because I wanted to create a ‘real user experience.’ I didn’t use my company account. I made a new CloudFlare account, added this domain, and started there. I didn’t want Multisite because that’s where my store is, and if I screw up the busy site, I don’t lose money or risk people’s information. While I’m sure it can be used safely, I knew I was going to experiment a little, so I wanted to protect myself and them

    Turn on CloudFlare, Turn off PageSpeed

    You heard me. I turned PageSpeed off for the domain I’m testing on. I love PageSpeed, but after talking to some people, I’ve been wondering about how well it handles things. Also with SPDY and HTTP/2, compressing HTML is less and less of a concern. I wasn’t sure if there was a benefit to having everything be filtered and compressed before it loaded. Was I making the experience worse? After making sure I still had mod_cloudflare active and up to date, I used .htaccess to turn off PageSpeed.

    Break Your Code Flow

    I need to point out that CloudFlare warns you about this one. They tell you that if you need to SSH, you should use a different record or the IP. That wasn’t a big deal. I’d kept ftp.example.com separate for FTP anyway. All I had to do was change my SSH aliases to point there as well.

    I forgot that I use Git on my own server. I wanted to update a script and Coda hung. It took my brain a moment to remember that I was using Git with SSH so I had to remember how to change the remotes on Git:

    $ git remote -v
    # origin  me@example.com:USERNAME/REPOSITORY.git (fetch)
    # origin  me@example.com:USERNAME/REPOSITORY.git (push)
    

    This was a problem. My example.com main domain hit CloudFlare. So I had to change that to use ftp.example.com as well. That wasn’t too hard, just running this 6 times.

     $ git remote set-url origin me@ftp.example.com:USERNAME/REPOSITORY.git
     $ git remote -v
     # origin  me@ftp.example.com:USERNAME/REPOSITORY.git (fetch)
     # origin  me@ftp.example.com:USERNAME/REPOSITORY.git (push)
    

    Still, I felt pretty silly!

    Break Your Email

    Same song, second verse. I had to add in a CNAME for smtp.exmaple.com on CloudFlare because I use that to send emails and I use mail.example.com to receive. By default they know mail should be ignored. The scan didn’t pick up smtp. I forgot about it until I tried to reply to an email.

    Break Your Tools

    Not the ones on my computer. My other CMSs broke. CloudFlare is used to WordPress, and has Five Easy First Steps to using WordPress and CloudFlare.

    In that document, they specify this:

    Create a Page Rule to exclude the wp-admin or wp-login sections from CloudFlare’s caching and performance features. You can access PageRules in your CloudFlare ‘Settings’ options.

    e.g.

    *example.com/wp-admin/*
    *example.com/wp-login/*
    

    Why do this?

    While there is not always an issue, we have seen instances where optional performance features like Rocket Loader may inadvertently break certain functions (editors, etc.) in your WordPress back end.

    Except there are major problems (besides the fact that the second example should be *example.com/wp-login* without the trailing slash)! The free version only gives you three rules. I’m using four apps (WordPress, ZenPhoto20, Yourls, and MediaWiki). That means if I need to white list all of them, I’m out of luck.

    Then there’s the problem that *.example.com matches blog.example.com and www.example.com but does not match example.com and guess what? I’m using example.com without the WWW. I hate WWW. And yes, you can use a naked domain with CloudFlare.

    Break The Site (For One Person)

    Someone pinged me to let me know the site was down in Scotland:

    Website is Offline Message from CloudFlare

    It wasn’t in Manchester or Dublin. It wasn’t in the US. It wasn’t in Canada. I opened a support ticket after making sure that it wasn’t really me. The server was up (all other sites, including this one were up) and I could get the site via anonymous proxies. Only that user had an issue, and I was 100% positive I had whitelisted everything in CSF (it’s the same thing I do for Jetpack). But the website claims a 522 is my server.

    This was never resolved, and demonstrates a major issue in the process. The user was a non-web savvy user. She shouldn’t have to be, though. She just wanted to visit a site and read things. It was very annoying.

    Drop Server Load

    Okay. So this part worked.

    Graph showing Server load on day 3 leveled out

    It only shows up starting day three because I didn’t flip DNS over right away. I had to turn off PageSpeed, upgrade PHP, make sure nginxcp was going to work with it… There was prep work. Day three, the little spikes vanished. You still get big ones because that’s when the server runs backups and upgrades. The little spikes are, normally, when I have a new post on the site.

    There’s been no perceptible change to bandwidth. All other sites on the server are, however, notably faster.

    End Result?

    It looks like CloudFlare worked. The stats say I’m using 25% less CPU with CloudFlare, which is interesting, but now that it’s baked, I want to try something else, just for grins and giggles.

  • TimThumb and the Pseudo (D)DoS Effect

    TimThumb and the Pseudo (D)DoS Effect

    Over the course of a day, my server rebooted httpd twice. That’s not a common thing for me, as after days and hours of work, I managed to finegal my server into a semblance of stability so it could handle massive loads. At first I thought it was due to my traffic spike of an increase of about 1500% (no I did not screw up a decimal place there, my traffic went from a couple hundred to nearly 4000 in one day). Then I thought that couldn’t be right, because my traffic had actually mellowed out by the time I was crashing.

    So I went to my emails and pulled up the Apache Status log and noticed that 70% of the calls to my site were GETs to pages like this:

    /wp-content/themes/newsworld-1.0.0/thumb.php?src=/g0../0d1.
    /wp-content/themes/DeepFocus/tools/timthumb.php?src=/g0../0
    /wp-content/themes/TheProfessional/tools/timthumb.php?src=/

    Thumb Wrestling And thanks to that massive spike in traffic, my server was slowing down to the point that HTTP was becoming unresponsive and it had to reboot itself. In short, the TimThumb exploit was causing my server to behave like it was under a Denial of Service Attack, even though I don’t use TimThumb! My server was able to handle this, but if I’d been back on my old Shared Server, I’d probably have not gotten a text from the server at 11pm saying “Man, we had to reboot, but it’s okay now. Can I have a beer?”, but instead woken up to ‘Dude, where’s my website!?’ And this is with having a fantastic web host who cares and takes the time to help me out.

    Normally this is where I’d tell you what to do if you’ve been infected via the TimThumb exploit, but Cleaning Up the TimThumb Hack covered it pretty well. Just remember this, if you have been infected, you must reset all your passwords. This is true of any and all hacks. As soon as someone has access to meddle with files on your server, you could be hurt worse than you know. At the very least, you need to read the post “Technical details and scripts of the WordPress Timthumb.php hack” by the guy who ‘fixed’ TimThumb.

    What I wanted to do here was sort out how to block people who were looking for timthumb.php files (I can’t block thumb.php as I use that filename elsewhere). Reading up on Perishable Press’s Stupid .htaccess Tricks it’s clear we can do this:

    # prevent viewing of a specific file
    
     order allow,deny
     deny from all
    

    That should simply block access. An .htaccess block is a pretty simple way to reduce your server load, because people are getting punted before they get very far into things. Still, it’s something I have to install on each account on my server. Right now they’re just hammering ipstenu.org, and this is not the only domain on my server. This is, by the way, the same problem with using a plugin like WordPress Firewall. It’s a fantastic idea, if all you have is one account on a server. Protect yourself.

    I don’t. I run a VPS, and I have four domains here which I should be protecting. It’s easy enough to make that change on all four, plus two other external servers, but is that the best use of my time? I say no. I think I should automate as much of this as I can. What I really want is to say ‘If you’re looking for timthumb.php, then the odds are you’re looking for security vulnerabilities, and you need to just die in a fire.’ Or at least firewall block them. Seeing as I already have CSF on my server, it was logical for me to start there.

    Ice Blocks floating in the lake - Sarah OhBlocking an IP is easy, and I can do it via command line or GUI. Auto-detecting a URL though, is more complicated. Part of me thinks that, much like I can auto-block someone who tries to log in with any ID 10 times in a row, I should be able to add in a file pattern somewhere. Turns out you can’t, at least not the way I wanted. Instead, you have to do it differently.

    TimThumb’s exploit scanner isn’t actually a DDoS Attack, but it acts like one. A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make your site unavailable. Basically they hit your site with so much traffic that it’s incapable of its job: serving up webpages to happy readers! That’s why I call this a pseudo (D)DoS attack. The goal of the scanner is to see if you’re using Timthumb and, if so, to be evil. It’s not really distributed (i.e. multiple computers at once), though because of the number of people running the exploit scanner, it can seem that way. The side effect is that your site is hammered and can’t do what it wants. Which leads us to Connection Tracking.

    CSF has a tool called ‘Connection Tracking’ which lets you limit how many times a specific IP can hit your site at once before they get tossed to a shit-list. I set this to 300, and told it to only scan ports 80 and 443 (because I need to have unlimited FTP, and sometimes I log in from different IPs – yes, my home IP is on the whitelist).

    Connection Tracking. This option enables tracking of all connections from IP addresses to the server. If the total number of connections is greater than this value then the offending IP address is blocked. This can be used to help# prevent some types of DOS attack.

    Care should be taken with this option. It’s entirely possible that you will see false-positives. Some protocols can be connection hungry, e.g. FTP, IMAPD and HTTP so it could be quite easy to trigger, especially with a lot of closed connections in TIME_WAIT. However, for a server that is prone to DOS attacks this may be very useful. A reasonable setting for this option might be around 300.

    Setting this up is a little less obvious for the new person. Go to WHM > Plugins > ConfigServer Security & Firewall and look for “Firewall Configuration”
    CSF Firewall Configuration

    Click on the button, look for CT_LIMIT and change it to 300.
    CT_LIMIT settings

    Scroll down, click ‘Change’ and then restart CSF.

    Now, you could put this as low as 100, or as high as you want, but I did some reading and 300 seems like something not too likely to trip innocent people up, but just enough to catch the bad guys. I may way to lower this to 200 or so, but I know that a lot of people come to one part of my server for the image gallery, and they tend to open lots of pages at once. I don’t want to hurt them. The other thing to keep in mind is how short is the block time. The IP block is set for 30 minutes, which isn’t much at all, but it could be just enough to make the transient DDoSers go away. ETA: As of February 2012, I’ve actually lowered this to 50, and it’s made a world of difference! My day to day CPU is a little higher, but the number of spikes that caused outages has dropped.

    I’m not doing this to stop the people who want to bring my server to its knees. I’m doing it to stop the people who are ‘scanning’ for exploits. A true DDoS is hard to block because as soon as I block it, I have to defend against it again and again. CSF would be like a sump pump in your flooded basement, always running until it burns out the motor. It comes from too many sources, and for the little guy (i.e. me), I may just have to shut things down for an hour and wait it out. But these scanners, well, I can block them with this trick, and not hurt my server doing so!