Save Bacon With ModSecurity

Earlier this month, my company DreamHost had a small snafu with ModSecurity. The tl;dr is that we had a typo and it stopped some people from being able to access or properly use Jetpack. Thankfully, the WordPress community (including everyone at Jetpack, whom I owe drinks and/or dinner) is filled with amazeballs awesome people who helped us figure out everything and sort out customers who, upon getting what appeared to be a Jetpack error, went there when they needed to go to DreamHost.

These things happen. Code isn’t perfect, people aren’t perfect, and everyone makes mistakes. Of course, on the internet it’s unreasonable to assume a legit gaff, and I’ve seen people call out “Why was DreamHost pushing out these tweaks?” and “Didn’t they test?” so I thought perhaps it was time to explain why we use Mod Security and why, even though it’s my nemesis, I like it a lot.

What is ModSecurity anyway?

ModSecurity (aka modSec or mod_sec) is an open source web application firewall (WAF). That means it sits between your website and the world, blocking all the bad people. When we have those brute force attacks, ModSecurity is key in blocking them. It blocks people who attempt code injection attacks like this:

http://www.example.com/wp-login.php?username=admin'">DROP%20TABLE%20wp_users--

Now that would never work on WordPress core, but the Bobby Tables attacks have the potential to kill your site if you have a plugin or theme that’s insecure. Most hosts have customized their rules to check for things like hitting the wp-login page improperly or passing through credentials directly. That means if someone tries to log in to your site without clicking the submit button (yes, you can code that), it will block them.

One of my favorite things about ModSecurity is that you can hook it into another firewall like ConfigServer, Fail2ban, or even the built in linux feature of IPTables, and block any IP that routinely trips your security rules.

So why was DreamHost monkeying with it anyway?

Every host is constantly monkeying with ModSecurity. As attack patterns grow and change, your host has to adapt. There’s a team at pretty much every host on the planet who watches logs, studies them, and improves the ModSecurity rules. Heck, we even share our rules with other hosts when the situation calls for it, like that Brute Force attack back in 2012. It was brand new, we were all surprised at the aggressiveness, and we quickly shared information.

On any given week, your host is creating new rules and testing them in their dev environment, or on specific real servers that are designated “Go ahead, blow me up.” After all, we all know nothing beats real-world testing. And if push came to shove and one specific site was being hammered, we may push an experimental rule to them before we’re done testing everything, because it’s that or your site is down.

We’re always working and improving. Security is a moving target after all.

How come a typo slipped through?

If you can find me someone who makes a 100% perfect product every single time, I’ve got a bridge for you to buy. We tested everything we could think up, and interestingly enough, that Jetpack error didn’t impact all Jetpack users! We have a test box, with Jetpack, and it worked fine there. Go figure.

But I’ve often said your website is a pretty snowflake. It’s unique, and what you do with it is different from what everyone else does. Things I have and do on this server and this domain are wildly different from my other sites on this same server! The need for the site is different, and what it uses is different, so what it does when it communicates with the world? Different.

I’ve had days where one domain is acting like a prat, but the others are fine. And I’ve sat there thinking “But it’s the same on these domains! They’re on the same server for God’s Sack!” only to realize that the usage pattern of the sites were very much not the same. And that takes everything longer to fix because you have to narrow things down over and over until you actually find out what the heck you did wrong.

I can’t even tell you this will never happen again because I’m pretty sure someone will make a mistake again sometime in the future.

Conclusion?

I wouldn’t run a site without ModSecurity, but there are options.

In February 2013, Zero Science Lab released a study comparing it to Incapsula and Cloudflare. While ModSecurity came out on top (though it was noted to be more aggressive and caused more false positives), Incapsula has been working hard to fix it’s issues. There was actually a Round 2: Incapsula vs Cloudflare study in October 2013, and in this one, Incapsula is the clear winner. Of note, you won’t get the WAF protection on either for free.

The studies say, to me, that if you’re master of your own domain and want the firewall on your server to run yourself, use ModSecurity. If you’re going to farm that security out to the cloud, use Incapsula. There are, of course, benefits to putting the firewall on the cloud, and the major one is that you’ll be spared high CPU since the processing of the naughty people is done on their server, not yours. But of course, if they go down, you’re at risk, so you should probably have ModSecurity anyway.

After all, your website is important, right?

Ipstenu (Mika Epstein)

I’m Mika (aka Ipstenu) and I’m complicated. I’ve been on the net since the days of dedicated terminals and Novation CAT modems and solved Missionaries and Cannibals (or Goats and Cabbages) before I was six, and blog about WordPress and technical/computer things at Half Elf on Tech.

See Full Bio