Tag: servers

Farewell Mothra, Hello Ogra

For some reason I name my servers after Godzilla monsters. I currently have two – ogra and gamera. Gamera is a new DreamCompute box I’m using to test things on, but ogra is this server.

Gamera

There used to be a server called Gamera before Gamera. Original Gamera was my very very first VPS. At the time, I had four websites, all with separate hosting plans, and I mathed out that it would be cheaper to combine them to one VPS and learn that.

Boy what a jump that was! This was my first experience with a VPS at all, but I think that, in retrospect, it gave me comprehension of the web in a new way. Before, I was just a user of the internet. After, I understood why so many sites behaved differently. Gamera was actually how I got to know Mike Schroder! We’d added ImageMagick to WordPress and he was an aficionado. I was trying to test some things and he pointed out I didn’t actually have it installed. Off to the races!

Mothra

As Gamera got long in the tooth, I needed to upgrade some software and realized that doing so wasn’t going to be possible anymore without an OS upgrade. Instead of that, however, I looked into a whole new server on a new system: the cloud.

This really just meant I had a more dynamically upgradable server. It was easy to add more memory or diskspace. I could spin up new clusters as needed (though I haven’t needed to yet), and it was bigger and faster and better.

I learned a lot on Mothra. Everything from memcached and nginx to varnish and SSL was done there. I’ll miss it.

Ogra

Of course time went by and I installed all sorts of shit on my server. And, eventually, I wanted to upgrade to the newest operating system and test it without downtime. This can be done, but I decided to build a new server on the latest and greatest OS and migrate my sites. Building out the server was easy, as I have a document called “How I Installed Shit On My Server” and I was able to use that to rebuild everything I needed. Some things had changed and got updated, but in general it was pretty much the same thing.

Since everything was on cPanel and WHM, transferring sites was incredibly painless. Where, five years ago, I had to ask for help, this time I was able to press the pretty buttons and do it myself. The most terrifying part was the DNS. What I did was move accounts over one at a time, starting with the smallest site and building up. Then I’d change my /etc/hosts file to point to the new IP and verify everything worked. As soon as the site was good, I changed it to the new nameservers (ns3 and ns4) and moved on to the next one.

This was only a problem when I ran into my Aunt’s website, which has the DNS over on Microsoft. A few emails later, I logged in as her and changed the setting (and saved her account information for the next time). Sadly, Microsoft doesn’t let you point a domain to a CNAME, you have to use an A record, which means IP and doing my migration this way required a new IP.

What Now?

Now I’m waiting on EasyApache4 and full support for Let’s Encrypt in cPanel.

9 May, 2016
CloudFlare Experiment Ends Weirdly

I ended up turning it all off for one reason only.

I keep getting a 522 error on cloudflare.com.

Now. I have a working theory that it happens when I’m hitting my own site a lot (be it for development or as recently, a lot of traffic I need to reply to), but what would happen is I got an error 522 on my sites. So I’d go to cloudflare.com to whitelist my IP, since their explanation of “This means your site is down” was wrong (site was up, I was ssh’d in at the time), and I’d get a 522 on cloudflare.com.

Let me roll back to August 1st.

That night I went to make some changes on my site to the CSS and, instead of turning on Dev Mode in CloudFlare, I did my thing with Git, pushed my changes, dumped the cache of that CSS file, and was prepared to smile at my glory. Nope! I got a 522. This was odd, since I was currently on the server via SSH and the load on the box was 0.3. Naturally I went to support.cloudflare.com to try and see if I’d missed some directions only to get a 522 there as well.

Track that for a second. I got a 522 on CloudFlare’s domain.

The possible answers I could come up with, since I couldn’t read any of their documentation, as either my IP was blocked or the cache server I was proxying through was down. Since I could log into the dashboard for my accounts, I went in and tried to guess how to whitelist my IP. I couldn’t find that, so I opened a support request:

I can’t access support.cloudflare.com because of 522s. My IP is 172.249.156.169 – Is there any way I can get whitelisted?

I got reply that it was the wrong place to ask, which I don’t think was correct. One of the solutions (per a Google cache of the support page I couldn’t access, hello) said that you could whitelist your IP to see if that helped. Cool. Except I couldn’t get the part of the page to load that told me where and how that was set.

So I asked for support with the dashboard via the dashboard support panel. Instead I got someone telling me I had to open a new ticket. And he was incapable of transferring my ticket or saying “Hey, you can’t access the right support place, let me make a ticket for you! Sorry about that.” It was akin to telling me to email them to tell them my email was down.

I fumed. And then I kept clicking until I found the place to enter my IP. I did and magically CloudFlare started working for me! I quickly went and opened a ticket to complain that I couldn’t have a ticket transferred (or made for me), and suggested this:

If someone’s logged in via the dashboard and they’re getting a 522 on ALL Cloudflare sites, it’s a logical assumption that something blocked them. But if I can log in, the odds are I’m really me, so that should get an auto-whitelist. If that isn’t possible, can it be detected and alerted? “Hey, we noticed your IP is blocked. Would you like to white list it, since you’ve logged in we can be reasonably sure you’re not an asshat?”

They replied with a standard ‘A 522 means…’ and told me to whitelist their servers on my server firewall. For some reason the email didn’t get to me, so I made a new ticket.

In this ticket, I had to wait until I got another 522 (end of August) and when I sent in my error ID and a screenshot, I was told this:

This was a timeout between our cache server and the origin server that hosts support.cloudflare.com.

I think he actually meant “Our cache server was down.” because at that time I couldn’t get to any CF hosted site until I rebooted my router and got a new IP.

I don’t really buy this, though, and I think their IP block is too aggressive. I would run into it all the time when I was at a hotel. I’d be reading comments on my own sites and get blocked. And every single time I got blocked, it was from all of my domains and cloudflare.com. Sketchy as hell to me.

When I added in the problem that ‘always up’ actually meant if your site was down they’d put up a CloudFlare page to apologize for the site being down, I decided to turn it off. It clearly wasn’t helping me as much as I’d hoped.

This isn’t to say CloudFlare is terrible and you should never use it, just that it proved to be too frustrating for me to want to use.

14 September, 2015
How Do You Solve a DB Like Maria?

I was talking to my friend James about upgrading SQL. If you didn’t know, upgrading SQL is a horrifyingly monumental thing, because there’s no way back except restore from a backup. Minor upgrades are generally painless, but the CentOS warning is as follows:

Upgrades to new major releases (the first two digits in the version string) are more involved because there is a substantial risk of data loss.

Data. Loss.

It’s scary when you consider doing it for yourself. It’s horrifying when you consider doing it for a few thousand users.

On top of that is the issue that MySQL is owned by Oracle and they’re not exactly known for being good stewards of OpenSource. Unlike many other Open Source projects, Oracle owns the entire copyright to MySQL. All contributions are done if the developer has signed a “contributor agreement” that assigns ownership to Oracle. This isn’t all that weird, to be fair. When I worked for The Man, that was basically how things worked and it made sense. The work I did for the company belonged to the company.

Where this is weird is that Oracle has said that about a GPL product, even to parts of it the company has not written. Why is that? It’s because all contributors to the code have to sign a “contributor agreement” assigning ownership of the copyright to Oracle, which is not alone in this. Sun before them used contributor agreements to get full source ownership, and many other projects do the same.

Now, James and I looked at the MariaDB vs MySQL compatibility doc and had a laugh.

tl;dr “For all practical purposes, MariaDB is a binary drop in replacement of the same MySQL version,” except for this long list detailing where you’re screwed.

Now when you get down to MySQL 5.5 and MariaDB 10, the issues become very minor and unlikely to cause you migraines, which is a relief, but that list sure is long and daunting.

I’m not yet running MariaDB because it’s an all-or-nothing move. I can’t keep on MySQL, and I have a few old (ancient) bits of non-WordPress code on this server. I always stress that WordPress is not the limiting factors in server upgrades, and it’s still the truth.

I’ve started doing the recon work to make sure MariaDB will work for all situations on my server, for all apps, and I’m currently pretty sure that I’ll be fine, but I do have one way-out-there app to check into. They’re also one of the few people who pay me for hosting, so we may have to have a sit-down anyway to discuss their future.

@Ipstenu Yep!

— Gary (@GaryPendergast) March 24, 2015

The most important question has been answered.

8 April, 2015
Mailbag: SNI Incompatibility?

Kim asks:

You wrote an article which does a great job of explaining a number of things. My only question (comments appear to be closed so I could not post there) is the SNI – do you find that there are many people using browsers that are old enough that the SNI creates a problem? I have looked over the list of incompatibles and it does not seem to be that much of a risk, but I thought you might have more concrete information since you’ve been using the setup.

This relates to how I set up my SSL certificates, which is to use Server Name Indications and have multiple certs on one server with one IP. And the question is “Do we care about the old browsers?”

Let me quote my coworker.

IE8 is EOL, XP is EOL. We can’t support things forever.

XP makes up most of the sites that have issue with SNI so I’ve only found 0.006% of my visitors impacted.

Yes, I did that math properly. I checked it a couple times.

No. I’m not worried about SNI and I don’t care. We can’t support old things forever.

27 March, 2015
GeoIP Options
Thanks to crazy thinks like the EU VAT laws, sometimes we really have to know where people are coming from when they visit our sites. The problem with this is … how?

There’s a cool extension for PHP called GeoIP, which I’ve finally installed on this server (along with my upgrade to PHP 5.5 and some other things, yes, still on Apache, shut up Otto). The extension comes from MaxMind, who also have a pure PHP version you can use. I’m not because the GeoLite2 databases are distributed under the Creative Commons Attribution-ShareAlike 3.0 Unported License and that means I can’t include it in a WordPress plugin.

But that really made me wonder why it was okay not to attribute Maxmind when I used it via Pecl. I mean, technically I should, right? But where and how? I ended up putting a note in my site footer, to say that the site used the Maxmind DBs, but I haven’t included any note about that in my plugin since the DBs are included in the plugin, just called if the functions are found. It’s on you to install and attribute as needed.

Installing mod_geoip

Installing this is simple, from a server admin perspective.

Since you can’t use the yum install on Apache 2.4, I got to use a cPanel Custom Module, which meant running this:
```
wget http://easyapache.cpanel.net/optmods/custom_opt_mod-mod_geoip.tar.gz
tar -C /var/cpanel/easy/apache/custom_opt_mods -xzf custom_opt_mod-mod_geoip.tar.gz
```
And then I ran an EasyApache build. That was fine, I needed to do that anyway. Once that was done, I installed the pecl for GeoIP:
```
pecl install geoip
```
Done. Optionally you can add it to apache in either your .htaccess or (better) a conf file for your whole server:
```
<IfModule mod_geoip.c>
  GeoIPEnable On
  GeoIPDBFile /usr/local/share/GeoIP/GeoIP.dat
</IfModule> 
```
What about upgrades?

Every month you don’t upgrade your geoIP DB, the more your site sucks. Someone quoted a statistic that every month you don’t upgrade the DB, the accuracy drops by 1.5%. I can’t validate that, but I’d believe it.

Upgrades are fairly painless, thanks to geoipupdate, though it doesn’t include the IPv6 files for some reason. Still, being able to toss this into crontab makes my life easier:
```
38 15 * * 5 /usr/local/bin/geoipupdate
```
Of course… I did notice that there’s a new MaxMind DB Apache Module.

If you’re on nginx, you can grab the nginx geoip module too.

What if I can’t install PHP modules?

By request, I’d already added in the GeoIP2 PHP API to my wee little plugin. Not everyone can use mod_geoip or mod_maxminddb, after all, so it’s good to have options. And with this option, you have the question of how to update since geoipupdate won’t work anymore.

If you want to go hardcore, you can Auto-update your GeoIP databases with Cron via that very robust script. Or if you’re simple like me, it’s a geoip.sh script in your ~/scripts/ folder:
```
#!/bin/sh
cd /home/username/public_html/wp-content/edd-pec-geoip
wget -q http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.mmdb.gz
gzip -d -f GeoLite2-Country.mmdb.gz
```
And then I have this in my crontab:
```
30 22 2 * * /home/username/scripts/geoip.sh
```
Which is a lot easier for a lot of people.
19 January, 2015
The Road to PHP 5.x

If you use WordPress, you may be surprised to see that WordPress still supports PHP 5.2

PHP 5.2 was released in 2006, which is nearly a decade ago, and in the way that the internet ages, an impressively long time. In 2010, only five years ago, the last version of PHP 5.2 was released: 5.2.16. At that point, PHP no longer supported 5.2, but did apply security patches. True end of life hit in January 2011.

But back in 2006, when WordPress was getting off the ground, a lot of people were still using PHP 4. Not 5, 4. It wasn’t until mid 2010 that WordPress itself dropped support for PHP 4 (and MySQL 4). At that point in time, roughly 11% of WordPress installs used anything lower than 5.2 and this made it a pretty safe bet.

So why does WordPress still support PHP 5.2? Today if you look at the (not very accurate) WordPress stats, you can see that over 30% of people still use PHP 5.2

Back in 2010, WordPress was about 10% of the entire web. It’s double that today, so 33% of people on 5.2 is significantly larger than 11% back on pre 5.2, and with those numbers, it’s easy to see why it has to keep supporting PHP 5.2 … for now.

But the next obvious question becomes why is PHP 5.2 out there? And the answer is that WordPress is only 22% of the internet. Flip that around and remember that 78% of the internet is not WordPress. WordPress is not everything, nor should it be, so the webhosts of the world do have to consider than when they begin to upgrade.

Most major webhosts are in some state of killing PHP 5.2 with fire (seriously I am very excited for when it’s finally gone from DreamHost). When we upgraded people to 5.4, we found a lot of people who had very odd code out there that just didn’t work on 5.3 or 5.4, and upgrades broke them. We also found a number of people using WordPress who broke, mostly because they’d customized PHP 5.2 and forgotten, but also some who were on things like WP 2.x and were shocked, just shocked, it needed to be upgraded.

As developers, we want to be able to force everyone into a place where we can upgrade PHP (and WordPress) and have no compatibility fears. We want to use the new features of PHP to allow us to craft better, faster, more efficient code. We want to give users the features they ask for. But we can’t until everyone upgrades. And thus the vicious circle begins.

Would WordPress dropping support for PHP 5.2 speed up it’s demise? No. Not at all. Because WordPress is a drop in the ocean of the hassle that is upgrades. Do I think WordPress should drop support for PHP 5.2? Yes, but not the way you’re thinking. I would love to see WordPress stop supporting 5.2 in the sense that it should stop testing against it and worrying about backwards compatibility with PHP 5.2. It should check, on upgrade and install, that PHP 5.3+ is used and go from there. Heck, if it had a big alert “Hi, you’re on PHP 5.2, please upgrade!” on the admin page, that would be awesome.

But I don’t know that there are any PHP 5.3 (or 5.4 … or 5.5 or 5.6) specific features that absolutely require WordPress to be on 5.4 at this time. Frankly, that doesn’t matter at all because the issues with upgrading are far less related to where WordPress is going and more directly the cause of where servers have been. Most hosts grow organically, servers being built following a process and then (eventually) via an automated tool. But because of that, a lot of old servers and operating systems don’t lend themselves well to upgrades because they’ve been built rather … higgeldy-piggeldy one might say.

It’s that history, the drama of people not seeing the future 10 years ago, that puts many hosts in a position where upgrade is actually going to mean moving users to a new server with new features. And that is not something to be done lightly. We can’t just pick people up when we want and move them. There will be downtime, there will be outages, there will be delays. And because of all that, these moves take longer than you want.

This is not to say the hosts aren’t doing the right things, just that they take longer than anyone (especially the host) would like. And believe me. No one wants PHP 5.2 gone more than a web host.

5 January, 2015