Category: How It Works

The Internet is Down
When things were new and we used to dial in to a BBS on someone’s computer, what we meant by ‘the internet’ is down was pretty simple. Either our phones were down and we couldn’t dial out or someone else’s were down and we couldn’t dial in.

On August 12th, a network outage took my server ‘down.’ Now, trying to explain this on Twitter was complicated, since it’s a more than 140 character explanation. The situation was pretty basic. The internet pipe leading to and from my server wasn’t working right. But what did that mean?

As I love to do, let’s step back and think about all the various ways your ‘internet’ might ‘break.’ It’s a fun thought experiment, and this is in no particular order.
- Your home/work internet isn’t working and no one can get anywhere
- Your device’s internet connecter isn’t working and you can’t get any signal
- You’re in a place with no signal/wifi
- Your firewall is preventing you from accessing a site
- The server that houses site you’re trying to visit is offline (or on fire)
- The site you’re trying to visit has a code error and nothing loads
- The DNS is wrong for the site
- The nameserver is wrong/changing (mea culpa)
- The internet connection from the site to the rest of the world is down
- There’s a problem in between you and the site
That list is incomplete. What happened to me on the 12th was the last one, however, and it was caused by something particularly weird that can be summed up as this: We finally hit 512K BGP routes on the internet today and ran out of room.

https://twitter.com/TheProtestBoard/status/499270694702972928

Of course, what’s BGP is the next question. From Reddit

BGP is a routing protocol that advertises routes externally, each large organization advertises some BGP routes at the edge of their network. Each edge device has a routing table with all the advertised BGP routes from around the internet.

So think of it like a giant phone book, and we ran out of pages. Now before you get scared, not every router needs all the tables. Instead, most routers have the core ones everyone needs, and then they link out to other routers and tables for the rest. These tables act as giant maps for the entire Internet, and those maps are pretty damn big.

A lot of routers, especially Cisco which I think powers most of the Internet, simply started dropping routes when they hit the 512k limit. That means you simply could not get from point A to point B, or in this case, you couldn’t get to your website from your ISP. I could, for example, get to my site on my phone and from my home internet, but not my office. Go figure. The routers had no idea how to find my domain.

This isn’t something ‘new’ by the way. In may, the IPv4 routing table hit 500k routes and the prediction was we’d hit 512k no sooner than August, more likely October. Oops.

As Otto put it:

Everything was affected. See
http://downdetector.com/ for example. All those blue graphs should usually
be quite flat.

That’s AT&T. It was pretty much the same for everyone, though.

The fix? Well systems engineers spent their August 12th reconfiguring their routers and in many cases upgrading memory, but it’s a practical limitation of the Internet. That isn’t a long term fix, either. Nor is IPv6. Oh, I should explain that too.

Internet Protocol version 6 (IPv6) is the latest version of the Internet Protocol (IP). That’s your internet address, your IP. It’s possible to share them, like all my domains have the same one, and you can change them if you need to, but mathematically speaking there’s a limit to how many there can be, in addition to those routing tables. This gets worse when you realize that every single device on the Internet is assigned an IP address for identification and location definition. Your phone. Your iPad. You get the idea.

There are improvements to the mess with IPv6. We’re using IPv4 for about 95% of the net right now, and the IP ‘blocks’ you get take up a lot of room. But with IPv6 the blocks will be larger and store more, so they’ll paradoxically take up less room. But it’s not a full fix. We’re going to have to come up with a better way to store the data for the tables, because things are only getting bigger.

On the plus side, for the first time in a long time, when someone yelled “The Internet is broken!” they were actually right.
13 August, 2014
Mailbag: Multisite Theme Activation
Briany from Ireland has a cool idea (put the guilt away, Briany, I think this is a pretty wild concept and I like it). This is the sort of ‘support’ email I love because it’s not a yes/no answer, but a theoretical concept to think about and parse!

Here’s the meat of the email:

SO MY FASCINATING QUESTION IS THIS; In a multi site network is there any way to use 2 separate themes for each or any sub-site based on the URL used (or any other method) Example 1 Visit www.sitename.platform.com and view the whole site using theme A (Standard) Visit www.sitename.com and view the whole site using theme B (custom)

At first I thought “Briany can’t possibly be asking ‘How do I activate a separate theme per site.’” and then I realized the question was for the network! So to phrase it in WordPress terms, if I visit the site via foo.example.com, I get theme foo on every site on the network.

(If you just want Site A to have one theme, and Site B another, that’s easy. Either network activate the theme and select it on the site, or go to WP Admin -> Network -> Sites, click on edit, click on the themes tab, and activate the theme you want for the sites. Then go back to the site and select that theme. A couple steps, yes, but relatively easy.)

Theoretically yes, yes you can. There’s a plugin called Theme Switcher which lets users pick, and based on some code on StackExchange, you can change the theme based on users, but the issue here is that you want to only change it per domain for that user.

It’s certainly possible to change settings by detecting the domain, I do that with SSL.
```
if ( $_SERVER["HTTP_HOST"] == "foo.example.com" ) {
    // My Code Here
}
```
That’s the easy part. The hard part is keeping that setting when I go from foo.example.com to example.com…

I sat and kicked this idea around for a while. It would be much easier on a single install of WordPress, since I could make everything relative and then just use the host name. But you have to have a way to track the starter domain, and have it be per-visitor, which means you have to use cookies, and read from that, using setcookie() (which is a PHP thing, not WP specific).

At that point, I think I would close the book and say “No, it’s possible, but not a good idea.”

Why, you ask? Caching. How the hairy hell could I possibly cache that if the theme changes every time for every user? Maybe, maybe, I would do it with multi-networks, and define a theme per network, but not per-domain. Obviating any caching would pretty much kill my sites, and even a good opcode cache (I use memcache) will be usless in that scenario.

By the way, there are a lot of neato plugins to change themes based on weird things, like Domain Theme, which is great for single installs of WordPress.
11 July, 2014
Yum!
I’ve touched on it a couple times, and it’s related to why I love things like Homebrew, but I like package installers. While I can, and have many times, installed and packaged from source, it scares me and I don’t like it. When I talked about Homebrew, I mentioned in passing that I use yum to manage packages on my servers, and someone asked me “What’s that?”

When I started this blog in 2009, the very first post was about understanding how to mess with my VPS and tweak it to work well with WordPress and everything else. In the five years since, I’ve learned a great deal about servers, tweaks, how to break things, but also how to upgrade them smartly. I’ve had struggles with SVN and GIT, but I understand that managing versions and revisions is a sane way to handle upgrades. But at the same time, not compiling code means I have more free time to mess with the stuff I like.

This brings us to Yum.

Yum is a software package manager manager, which means it checks various RPM Package Managers, sees if there is software you have that has an update, and updates it. An RPM Package Manager (or an RPM, yes, it’s a recursive acronym, just like PHP) stores packaged versions of code… I have a feeling someone’s looking at me like I spoke a new language.

Let’s step back further. Your server is a computer and runs software. Most people have the experience of installing software via packaged installers (I used to make them for a living). When you download an app onto your phone, the phone downloads the package and runs the installer. This is similar to WordPress, right? You download the zip, unpack it, and run an installer. But your server, well, for a very long time people didn’t have installer packages, they had source code. The source code was downloaded, unzipped (hush, you know what I mean), compiled, and then installed.

Credit: xkcd comic “Compiling”

Yes, compiling code takes a long time. That joke is less funny than it is accurate to some of us. It’s not something most of us do any more, though, because code now tends to come pre-compiled, and that’s where package managers come into play. You see, someone realized that they could pre-compile code for all servers. It’s not the same code for all the servers, though, because there are so many flavors of servers, it’s mind boggling.

But that said, if you know your flavor of server, you can use an RPM that matches to install software. So the RPM is like a massive server that has all the available installs for your server. When you add in yum, which installs the packages, you can then enter a world of automation where every night your server checks for new packages and installs them!

Yum has a meaning. “Yellowdog Updater, Modified.” I didn’t say it was a good meaning. Yum has a bunch of obvious commands like “yum install NAME” and “yum update” which you can use to install extra add-ons like Memcached and so on. There are also yum utilities (yum-utils, which let you further customize automation by scripting commands.

Just to touch on one at random, today I got an email from my server saying that it had run /usr/bin/yum -c /etc/yum.conf -y update for me. This is normal, I configured it to do that at midnight every day.

There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.

Now it did run the rest of the installs, so I did what any smart person does. I went and looked up this new command, only to find a string of bug reports from 2007-2009. It’s 2014, so I went and ran it once. It cleaned up one package and said I had another 244 left. Interesting. I ran it again. 243. I saw my day flash before my eyes and then decided to run the safer version:
```
yum-complete-transaction --cleanup-only
```
Safer and faster. Everything was cleaned, and update runs great now.

Is there a risk with this automation? Of course! Which is why I take a backup every night, right before the update happens. I’m not crazy after all.
18 June, 2014
Mailbag! .htaccess questions
New thing! So many people email me for tech support, which I’m pretty clear on how you’re not going to get it. But Ken (the web mechanic) asked some pretty basic questions, and I’ve decided to answer some of them.

In public. Lucky you, Ken. Don’t worry! These were good questions. See, one of the (many) reasons I love WordPress and the support forums is that the answers are public so everyone can see what the question was and how it was answered! This is hugely important to foster a community, so that’s why I’m going to answer this in public, with your personal information removed, of course.

Ken’s basic concern is that .htaccess is confusing, and is there a preferred order? The answer is yes. The basic idea is that .htaccess rules are a top-down process. The server reads the file from the top on down, in order, 1-2-3. For this reason alone, it’s obvious why you don’t want a super long .htaccess file: more to read takes longer!

The WP permalink area… Should that always be dead last?

Yes! WordPress rules always go last. Remember what I said about top-down? If you were to put WordPress at the top, you would load WordPress, process it, and then do the rest of the rules. Which once you say that, it’s pretty obvious eh?

Deny IP addresses/ referrers. To me it would make sense for them to be at the beginning… Would that be true?

True! The access controls (including IP blocks) first. Redirects go next, starting from most specific (about-me to about) first to general last. Then your Rewrite rules.

Compression/Caching/mod_expires… I haven’t a clue where they most appropriately go. Securing wp-config, htaccess itself, other files, etc. … Before? After? the WP permalink block.

I put them before my re-writes. Since I use a deny to secure .svn type files, it’s an access control so it goes first.

So how does this work? Here’s a practical example. You want to do the following: remove www from your domains, protect your wp-config file, protect your comments and login from direct attacks, redirect some old pages from before you were WordPress, redirect your old permalink formats, and gzip/compress things. Oh and run WordPress!

The order would go like this:
1. Access Control: This is the part where we’re protecting specific files, but also blocking IPs. Basically it’s ‘Security First.’
2. Remove WWW: We want to make sure everyone’s redirected to the non-www page. If you’re redirecting specific domains (like I send tech.ipstenu.org to halfelf.org), you do it here as well.
3. GZIP: I do my compression here, though it woudl work just as well swapped out with the next one.
4. Redirect: Here we’re talking one-off redirects like sending ‘about-me’ to ‘about’.
5. Rewrite: The ReWrite rules are the ones where you say “Send http://example.com/2014/01/10/postname/ to http://example.com/postname/” with those rules with regex.
6. WordPress Rules: Last. Always always last.
And that is .htaccess!

If you want a look at how my .htaccess is structured, see My super-secret .htaccess file, which hasn’t changed much since 2013. I do a couple things out of order, but they’re minor enough. As long as I can limit any recursive loops with the .htaccess checks, I’m doing good.
7 May, 2014
Save Bacon With ModSecurity
Earlier this month, my company DreamHost had a small snafu with ModSecurity. The tl;dr is that we had a typo and it stopped some people from being able to access or properly use Jetpack. Thankfully, the WordPress community (including everyone at Jetpack, whom I owe drinks and/or dinner) is filled with amazeballs awesome people who helped us figure out everything and sort out customers who, upon getting what appeared to be a Jetpack error, went there when they needed to go to DreamHost.

These things happen. Code isn’t perfect, people aren’t perfect, and everyone makes mistakes. Of course, on the internet it’s unreasonable to assume a legit gaff, and I’ve seen people call out “Why was DreamHost pushing out these tweaks?” and “Didn’t they test?” so I thought perhaps it was time to explain why we use Mod Security and why, even though it’s my nemesis, I like it a lot.

What is ModSecurity anyway?

ModSecurity (aka modSec or mod_sec) is an open source web application firewall (WAF). That means it sits between your website and the world, blocking all the bad people. When we have those brute force attacks, ModSecurity is key in blocking them. It blocks people who attempt code injection attacks like this:
```
http://www.example.com/wp-login.php?username=admin'">DROP%20TABLE%20wp_users--
```
Now that would never work on WordPress core, but the Bobby Tables attacks have the potential to kill your site if you have a plugin or theme that’s insecure. Most hosts have customized their rules to check for things like hitting the wp-login page improperly or passing through credentials directly. That means if someone tries to log in to your site without clicking the submit button (yes, you can code that), it will block them.

One of my favorite things about ModSecurity is that you can hook it into another firewall like ConfigServer, Fail2ban, or even the built in linux feature of IPTables, and block any IP that routinely trips your security rules.

So why was DreamHost monkeying with it anyway?

Every host is constantly monkeying with ModSecurity. As attack patterns grow and change, your host has to adapt. There’s a team at pretty much every host on the planet who watches logs, studies them, and improves the ModSecurity rules. Heck, we even share our rules with other hosts when the situation calls for it, like that Brute Force attack back in 2012. It was brand new, we were all surprised at the aggressiveness, and we quickly shared information.

On any given week, your host is creating new rules and testing them in their dev environment, or on specific real servers that are designated “Go ahead, blow me up.” After all, we all know nothing beats real-world testing. And if push came to shove and one specific site was being hammered, we may push an experimental rule to them before we’re done testing everything, because it’s that or your site is down.

We’re always working and improving. Security is a moving target after all.

How come a typo slipped through?

If you can find me someone who makes a 100% perfect product every single time, I’ve got a bridge for you to buy. We tested everything we could think up, and interestingly enough, that Jetpack error didn’t impact all Jetpack users! We have a test box, with Jetpack, and it worked fine there. Go figure.

But I’ve often said your website is a pretty snowflake. It’s unique, and what you do with it is different from what everyone else does. Things I have and do on this server and this domain are wildly different from my other sites on this same server! The need for the site is different, and what it uses is different, so what it does when it communicates with the world? Different.

I’ve had days where one domain is acting like a prat, but the others are fine. And I’ve sat there thinking “But it’s the same on these domains! They’re on the same server for God’s Sack!” only to realize that the usage pattern of the sites were very much not the same. And that takes everything longer to fix because you have to narrow things down over and over until you actually find out what the heck you did wrong.

I can’t even tell you this will never happen again because I’m pretty sure someone will make a mistake again sometime in the future.

Conclusion?

I wouldn’t run a site without ModSecurity, but there are options.

In February 2013, Zero Science Lab released a study comparing it to Incapsula and Cloudflare. While ModSecurity came out on top (though it was noted to be more aggressive and caused more false positives), Incapsula has been working hard to fix it’s issues. There was actually a Round 2: Incapsula vs Cloudflare study in October 2013, and in this one, Incapsula is the clear winner. Of note, you won’t get the WAF protection on either for free.

The studies say, to me, that if you’re master of your own domain and want the firewall on your server to run yourself, use ModSecurity. If you’re going to farm that security out to the cloud, use Incapsula. There are, of course, benefits to putting the firewall on the cloud, and the major one is that you’ll be spared high CPU since the processing of the naughty people is done on their server, not yours. But of course, if they go down, you’re at risk, so you should probably have ModSecurity anyway.

After all, your website is important, right?
28 April, 2014
Very Voluminous Vagrants
I finally sat down and installed Vagrant.

Specifically I installed Varying Vagrant Vagrants. I read, like I do, and the directions were thankfully pretty clear for a change about what the prerequisites were, and I found myself with an install of Vagrant up and running within an hour. At this point I joked “Now what?”

Vagrant is yet another tool you can use to create test websites. It’s a super-powerful command line tool one might use instead of DesktopServer or MAMP. Now, I love MAMP. While I find DesktopServer more ‘flexible’ in that I can spin up a client’s site really fast to check it, MAMP is even faster to boot when I just want to see one site and test a plugin on it, in part because I use MAMP No Password and run it on port 80 (which lets me use Multisite).

But… Sometimes when I want to test, I want to test different configurations at once. Like I want to test on current (3.8.1) and trunk, and I probably want a Multisite Network as well. But Vagrant is way more than just a lot of different environments in one go. I mean, if that was all it was, I could edit my hosts file and make a couple extra sites in MAMP or DesktopServer and be done.

What do you get?

You get a lot with VVV, and if all that seems weird to you, it’s okay. The more I work on servers, the more I see a varity of weird setups and the more I want to mess with them for testing. This isn’t to say VVV is perfect. You can’t switch (easily) between PHP 5.2 and 5.3 and 5.4. There’s a whole discussion ticket on the matter.

Vagrant makes multiple servers. Not PHP/Apache instances. Servers. They’re virtual, but let’s be honest, so is the server this site is running on. That means I can mess with the server, install extra features if I need to test a clash with PHP version or add ons or whatever, I can do so without blowing up my laptop (or a real server) pretty quickly. That means my plugins test site (on a live server) may not be needed anymore, which led me to the first actual thing I had to do with Vagrant: Make my own sites.

My own sites!

You get five ‘default’ sites when you use VVV:
- http://local.wordpress.dev for WordPress stable (/www/wordpress-default)
- http://local.wordpress-trunk.dev for WordPress trunk (/www/wordpress-trunk)
- http://src.wordpress-develop.dev for trunk WordPress development files (/www/wordpress-developer/src)
- http://build.wordpress-develop.dev for a Grunt build of those development files (/www/wordpress-developer/build)
That’s four, I know. The fifth is “vvv.dev” which lists the other four sites. That’s pretty much what most of us need when testing. A local site, one on trunk, one using src, and one with grunt for those CSS/JS things.

I needed more:
- http://multisite.dev Subdomain Multisite (/wordpress-musubdomain)
- http://local.multisite.dev Multisite subsite foo (/wordpress-musubdomain)
- http://bar.multisite.dev Multisite subsite bar (/wordpress-musubdomain)
- http://baz.multisite.dev Multisite subsite baz (/wordpress-musubdomain)
- http://folder.multisite.dev Subfolder Multisite (/wordpress-mufolder)
- http://local.plugins.dev Plugins tests (/wordpress-plugins)
Those all run stable. I though about naming them wordpress-something, but I needed the subdomains and it was easier to just make multisite.dev for that. I know what it is, and this is for me first. The way you do this is with Auto Site Setups, where Vagrant looks in your www folder and if it finds more folders with the right files, will build sites! Natually that means I made a little Github repo for this called VVV ASS. Because you’ve met me before, right?

What do you do with it?

Test all the things, really. Bang on plugins and code that isn’t right, make patches, and have a generally clean environment to play with. It’s also good for building on (core or plugins or themes).

I’ve been struggling with PHP Unit Tests, as well. I understand them, but I don’t really grok them yet. While I dig the idea of unit tests, creating them is a little complicated to me. Suffice to say, I know WordPress on push day wants to unit test all the things, so I should get used to this.
1. In terminal, navigate to your clone of the VVV repo.
2. Run vagrant ssh to connect to the virtual machine.
3. Run cd /srv/www/wordpress-develop and go into the folder.
4. Run phpunit
And there you go. What did all that do? A lot. Check out the Core Handbook on Automated Testing. Beyond that, I cannot tell you. Yet. Expect more on that later when I’ve gotten someone to explain it better.

Do I have to use VVV?

Nope! There’s VIP Quickstart, by Automattic (yes, that Automattic) which lets you bring up a WordPress.com-esque site. And there’s WP Vagrant which gives you more testing environments, so when you want to blow up core on PHP 5.2, you can do it easy as another vagrant up.

Am I sold?

Developer Ipstenu is sold. User Ipstenu still likes MAMP best. “Yes, I’ll fix your site” Ipstenu likes DesktopServer. Each has their own place. I’m so comfortable in CLI now, it’s more natural to use MAMP and VVV. DesktopServer comes in to play for me when I need to spin up a site specifically to test an existing site someone else broke.
4 April, 2014