Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: cdn

  • The Curious Case of a Comatose Cloud

    The Curious Case of a Comatose Cloud

    The summary here is that remotely hosting SVGs caused a massive slowdown.

    Isn’t the Cloud Magic?

    Nope. Well. No. It is totally magic, in that they’re great for large files and are an inexpensive storage alternative. But the cloud is, at the end of the day, just another server out there in the world, holding your data. Unless that cloud server is behind a CDN, you may not see a great deal of speed improvements on your site.

    And in my case, it didn’t.

    Diagnosing the Problem

    In building out a dev site with Tracy (LilJimmi), we noticed certain pages were really slow to load. 35 seconds slow. That’s unacceptable. I compared it to the live site, and it was faster, but some specific pages were still incredibly slow. What pages? The L Word for the most part. And as we inched closer to being done, I said “I’m going to fix this speed stuff before the weekend!” because you can’t go-live on a slow site. You just can’t.

    Once I was home, I fired up a local site, installed Query Monitor, and had a serious sit down with everything.

    It Wasn’t What I Thought

    My initial thought, the one I ruminated on during my bike ride home, was that it was the database queries. Most shows have one or two queer characters, but The L Word has 60 right now. While I may joke about wanting nine more, it’s a weird situation where I need to get the number of characters on the show without knowing the number of characters on the show. My assumption was that it was my calculation loop that caused the issue. That I was querying the queers too many times.

    Turned out it wasn’t (just) the number of characters, it was the number of tags.

    How It Got So Slow

    Most of the issue is my fault. Every single tag has a custom image associated with it. These images are stored remotely, in the cloud, and called as part of the design. The issue was that when calling the images I ran a check “Is the service available?” and if not, it would stop. When you make one or two calls, it’s no big deal. When you make a couple hundred, it adds up.

    The L Word had 2 icons per 60 characters, and then 15 tags, and then 30 more icons.

    Remote Get Is Slow

    I used wp_remote_get to process my images, and it was taking between .1 and .4 seconds per image. That adds up. At first I simplified my check-if-exists routine and more than halved the time to load from 35 to 15 seconds. But in order to drop the page back to 1 second-ish load times, I had to put the images local again. No matter what I did, if I loaded them remotely the best I could get was a 13 second page page load.

    Sometimes, local is better.

    What’s the moral?

    Obviously the moral is test before you rollout. Which we certainly did! By using Query Monitor, I was able to narrow down the speed for the database queries as well as the speed for all the HTTP requests. In doing so, I lost a CDN but gained speed. I’m trying to figure out how to speed up the CDN, maybe by finding a different front-end proxy, but right now I’ll keep using it for the large files like videos rather than the hundred small ones.

    I think it it’s worth it.

  • Housing Large Media Files

    Housing Large Media Files

    For the most part, the WordPress media library is fine. It falls down when we start needing to upload large files, though, for a variety of reasons. When we look at files like large PDFs or movies or podcasts, it’s really not a great solution to upload through WordPress itself. It’s slow, it’s clunky, and worst of all, those large file downloads can slow your site.

    The ‘right’ fix is to offload large media to servers that are built for this sort of thing. And in this case, I’m talking about Amazon AWS or DreamObjects.

    Of course, if you search for solutions like this, you’ll be disappointed. You will mostly find plugins that are geared towards syncing your media library with the cloud services. To be honest, the more I think about doing that, the less I feel like it’s a sustainable idea. Unless the CDN is super fast, it could actually make your site worse off by adding another domain to download from.

    I Don’t Trust Simple CDNs

    I’ve always been skeptical of CDNs in general. When there’s a shared library, it makes sense for everyone to call the same library. That keeps the world in sync. But your own media? The reason a CDN is good is that you can distribute your content across multiple locations. Provided you can actually, you know, do that. And keep them all in sync.

    Before hosting, I worked at a bank, and one of the headaches we had was pushing software updates across multiple servers and locations. After all, you can’t just upgrade the Chicago servers and not the LA and Atlanta ones. Plus you have to do them all at the same time, or make sure Jane in Idaho isn’t in the middle of depositing money when we reboot her server.

    Knowing how crazy all that is, I worry about keeping data in sync across all the servers. What happens when media is updated? Is the CDN built so that my primary location properly triggers updates for everything else, and the data is updated? No matter what, I’m sure I’ll end up with some data out of sync for at least a little while.

    In short, CDN synchronization isn’t simple and anyone who tells me it is, is selling something.

    So Why A CDN At All?

    Big files.

    The goal of a CDN is to speed up delivery of content without slowing down your website. For most images on a website, this isn’t a huge issue. But for those big files, it sure is. And uploading them to the could means three things:

    1. No lost disk space
    2. No lost bandwidth (if someone’s watching a movie for example)
    3. No lost speed (see the aforementioned movie)

    The rest of your CDN ‘needs’ can be handled properly by caching. I prefer server side, but as you like it. This means if I upload my large files to the CDN, I can link directly to them in my post content. Everyone wins.

    Except Uploading Sucks

    The common solution is to manually upload the file via a client like Cyberduck or Transmit, copy the URL, and then paste it into a blog post. Yuck. What I need is a file manager for the cloud. And that doesn’t seem to exist for WordPress.

    So I made something. DreamHost Objects Dropzone lets me upload files to DreamObjects, through WordPress, without touching the file server at all. It’s not perfect. It can be slow when trying to get stats on all the items in a bucket, and I don’t quite have an interface to make it easy to insert links and content into posts. Yet.

    Something to look forward to though.

  • Cookie Free Domains and CDNs

    Cookie Free Domains and CDNs

    After I dropped Cloudflare I went back looking at other ways to speed up my site and what could I optimize things. As I mentioned in Static Content Subdomain, one of the benefits of that mess was to allow me to have a cookie free domain and, in having a cookie free domain, I basically made a domain that could be used as a CDN for my (mostly) static files.

    It was noted then that, if you’re a non-www person, you have to actually use a separate domain. I do. I hate www in URLs. So I picked up a free domain for the site where I’m doing all this: ex-static.com. I probably could have gotten it even shorter if I’d tried sitecdn.net but that was short enough for me.

    Since I already had everything over at /home/user/public_html/static I created an add-on domain in cPanel and told it to use that as its home.

    Then I ran a search/replace on WordPress:
    wp search-replace static.example.com/wordpress ex-static.com/wp-content/uploads
    wp option set upload_path /home/example/public_html/static/wp-content/uploads

    Since I’d already changed my media settings, that was all I needed to really run a replace. It changed my media settings too! Since I was planning to move my static content, not just images, I changed the path too, so I had to go back and change upload_path as well, but wp-cli is my friend.

    Next up, because of the static content move, I copied my wp-content folder over and added this to my wp-config.php:

    define('WP_CONTENT_DIR', $_SERVER['DOCUMENT_ROOT'] . '/static/wp-content');
define('WP_CONTENT_URL', 'http://ex-static.com/wp-content');

    Boom. Everything’s where I want it.

    Sadly, it got messy for MediaWiki and ZenPhoto. The later doesn’t allow you to move your themes folder at all so I made a symlink to /home/example/public_html/static/themes/themename/ and called that a day. MediaWiki was a damn dirty lie.

    Supposedly you can set this:

    $wgStylePath        = "http://ex-static.com/wiki/skins";
$wgStyleDirectory   = "/home/example/public_html/static/wiki/skins";

    As soon as I did that, I got errors all over the place and had to change the directory path back:

    $wgStyleDirectory   = "$IP/skins";

    Weirdly named, $IP is set as follows:

    $IP = "/home/example/public_html/wiki";

    I did the same thing as with ZenPhoto and made a symlink then I could change the style directory.

    Lastly (and this was last for a reason) I changed my .htaccess rule:

    # CDN
<If "%{HTTP_HOST} == 'example.com' ">
        RedirectMatch ^/static/(.*)$ http://ex-static.com/$1
</If>

    This was last so that I could not break things mid-flight. And I added in this:

    # CDN
<If "%{HTTP_HOST} == 'static.example.com' ">
        RedirectMatch ^(.*)$ http://ex-static.com/$1
</If>

    Everything works as expected. And? I have cookie free domains and I’m ready to move all my data off to a CDN whenever I’m ready for that step. The CDN would only have to cover the ex-static.com domain, to boot!

  • CloudFlare Experiment Ends Weirdly

    CloudFlare Experiment Ends Weirdly

    I ended up turning it all off for one reason only.

    I keep getting a 522 error on cloudflare.com.

    Now. I have a working theory that it happens when I’m hitting my own site a lot (be it for development or as recently, a lot of traffic I need to reply to), but what would happen is I got an error 522 on my sites. So I’d go to cloudflare.com to whitelist my IP, since their explanation of “This means your site is down” was wrong (site was up, I was ssh’d in at the time), and I’d get a 522 on cloudflare.com.

    Let me roll back to August 1st.

    That night I went to make some changes on my site to the CSS and, instead of turning on Dev Mode in CloudFlare, I did my thing with Git, pushed my changes, dumped the cache of that CSS file, and was prepared to smile at my glory. Nope! I got a 522. This was odd, since I was currently on the server via SSH and the load on the box was 0.3. Naturally I went to support.cloudflare.com to try and see if I’d missed some directions only to get a 522 there as well.

    Track that for a second. I got a 522 on CloudFlare’s domain.

    The possible answers I could come up with, since I couldn’t read any of their documentation, as either my IP was blocked or the cache server I was proxying through was down. Since I could log into the dashboard for my accounts, I went in and tried to guess how to whitelist my IP. I couldn’t find that, so I opened a support request:

    I can’t access support.cloudflare.com because of 522s. My IP is 172.249.156.169 – Is there any way I can get whitelisted?

    I got reply that it was the wrong place to ask, which I don’t think was correct. One of the solutions (per a Google cache of the support page I couldn’t access, hello) said that you could whitelist your IP to see if that helped. Cool. Except I couldn’t get the part of the page to load that told me where and how that was set.

    So I asked for support with the dashboard via the dashboard support panel. Instead I got someone telling me I had to open a new ticket. And he was incapable of transferring my ticket or saying “Hey, you can’t access the right support place, let me make a ticket for you! Sorry about that.” It was akin to telling me to email them to tell them my email was down.

    I fumed. And then I kept clicking until I found the place to enter my IP. I did and magically CloudFlare started working for me! I quickly went and opened a ticket to complain that I couldn’t have a ticket transferred (or made for me), and suggested this:

    If someone’s logged in via the dashboard and they’re getting a 522 on ALL Cloudflare sites, it’s a logical assumption that something blocked them. But if I can log in, the odds are I’m really me, so that should get an auto-whitelist. If that isn’t possible, can it be detected and alerted? “Hey, we noticed your IP is blocked. Would you like to white list it, since you’ve logged in we can be reasonably sure you’re not an asshat?”

    They replied with a standard ‘A 522 means…’ and told me to whitelist their servers on my server firewall. For some reason the email didn’t get to me, so I made a new ticket.

    In this ticket, I had to wait until I got another 522 (end of August) and when I sent in my error ID and a screenshot, I was told this:

    This was a timeout between our cache server and the origin server that hosts support.cloudflare.com.

    I think he actually meant “Our cache server was down.” because at that time I couldn’t get to any CF hosted site until I rebooted my router and got a new IP.

    I don’t really buy this, though, and I think their IP block is too aggressive. I would run into it all the time when I was at a hotel. I’d be reading comments on my own sites and get blocked. And every single time I got blocked, it was from all of my domains and cloudflare.com. Sketchy as hell to me.

    When I added in the problem that ‘always up’ actually meant if your site was down they’d put up a CloudFlare page to apologize for the site being down, I decided to turn it off. It clearly wasn’t helping me as much as I’d hoped.

    This isn’t to say CloudFlare is terrible and you should never use it, just that it proved to be too frustrating for me to want to use.