Half-Elf on Tech

Thoughts From a Professional Lesbian

Category: How It Is

Making philosophy about the why behind technical things.

  • Why doesn’t the WordPress auto-upgrade work?

    This came up recently with the WordPress upgrade to 3.0 (and subsequent 3.0.1 bug fix). A lot of people had problems upgrading from WordPress 2.9.2 to 3.0, and for the most part, the fixes were simple.

    There was a known ‘memory bug’, where WordPress didn’t request enough memory from the server when upgrading and, since 3.0 is a bit bigger than 2.9.2, it failed. This was fixed with the plugin Memory Bump. Once you upgraded to 3.0, you could remove this plugin, because the new WordPress version took care of it on it’s own. There were other issues, though most of them seemed to be plugin conflicts or people using weird versions of PHP. I even had a note I kept handy:

    Memory Issues
    Per naicin

    We’re in the process of pushing out a plugin for this: http://wordpress.org/extend/plugins/memory-bump/

    In the meantime, you can also add this to your wp-config.php file:
    define('WP_MEMORY_LIMIT', '256M' );

    The issue is that WP *may* need more than the default 32 MB to upgrade to 3.0, due to the increased package size over 2.9. This is a “known issue”.

    3.0 handles this by always bumping you to a very high memory limit in the admin area, so you won’t see this once you’ve hit 3.0.

    Weird Errors and blank screens
    Rename your plugins folder to plugins-old (via FTP or SSH) and see if that fixes it. If so, name the folder back to plugins and reactivate your plugins, one at a time, testing between each copy.

    Believe it or not, those two answers were probably 90% of what I posted on the WordPress forums for the better part of a week. That and ‘Yes, these plugins don’t work on 3.0 yet’ (and I had a list for that too!).

    Then came Zarathustra — I mean WordPress 3.0.1 — and people began to have new issues. The most common was something like this:

    Downloading update from http://wordpress.org/wordpress-3.0.1.zip…
    Unpacking the update…
    Verifying the unpacked files…
    Installing the latest version…
    Could not copy file.: [some file]
    Installation Failed

    The file would be different for pretty much everyone.

    As far as I know, we’ve not found a fix yet, however I pointed out that when the auto-upgrade fails, you should try installing the update manually. It’s not that hard. It’s basically copying files to a server and if you can’t do that, I don’t think you should be running your own self-hosted site. This is why I run the techside of a site for someone else. Know your limits, I always say (my limits self-tests haven’t always been non-destructive, by the way).

    In this support post for the failed install, riddle said:

    I’d like to get back to the original poster’s point: 3.0.1 automatic upgrades are too unreliable for the very WP admins who most need one-click upgrades.

    ipstenu’s reply (“This is a problem with your server setup, most likely, not WP. The most likely reason would be that your PHP setup doesn’t like the way WP 3.0.1 is unzipping and copying files.”) misses the point. If there are common PHP configurations under which the automatic upgrade won’t work, then at a minimum the upgrade script should test for them before breaking the site.

    And I replied:

    You just hit the nail on the head as to WHY automagic upgrades fail.

    There aren’t config standards for PHP and server software that are across the board on every server known to man. Each company has their own settings and standards, and even then (as I pointed out to a coworker yesterday), unless you clone a machine onto the exact hardware, you’re still going to have an inexact copy.

    Servers are snowflakes, man. It sucks, but that’s it.

    WordPress is able to say ‘Your server must be able to do these things to RUN WordPress’, but even then, someone may have a wildly weird setup where stuff works poorly. It’s the nature of the beast and it’s impossible to test everything.

    If the automatic upgrade fails, do it manually. It’s not that hard (copying files). Heck, it’s easier than installing manually the first time!

    And really this is the problem with all software, web or otherwise. There’s a reason software has ‘minimum requirements.’ It’s the bare minimum someone tested on and the software worked. WordPress says ‘To run this software, you need PHP 5.2 and MySQL 5.0.15’ (as of of 2011, so get on that now!) and that’s pretty much it. That’s all you need to have WordPress run, but that’s not all you need to do everything you could possibly do with WordPress, and this includes those pretty auto-magical upgrades.

    For what it’s worth, I come at my viewpoint from someone who spent 3 years doing application testing. I had to make sure all the apps used together worked together on every computer. Which is impossible. There’s no way I could promise that every app someone might use will always work on a computer with every other app. To make our goal realistic, we ‘certified’ applications, and would list any known conflicts, as well as everything we knew it worked well with or needed tweaks.

    Knowing that, and knowing that servers are setup with just as much variation as desktops, means that when someone says that WordPress should have standards it works on I think they’re just naive. There’s really no such thing as a ‘standard.’ There’s no standard location for temporary files, there’s no standard location for a home folder, and there sure as hell is no standard setup for installing PHP on a Red Hat box. There are variables galore that could cause all sorts of issues. There are application conflicts, configuration miss-matches, setup issues and, of course, the fact that there are so many different types of ‘standard *nix servers’ that are used, with hundreds of patches and tweaks, that the target’s moving so fast, I think it just hit you in the head.

    So why doesn’t your automated upgrade work when you click the button on WordPress? The possibilities aren’t endless, but they’re pretty thick. You’re better off learning how to do the manual upgrade if you have wild errors no one’s seen before.

    And I hold by my belief that if you can’t perform a manual upgrade, you need to rethink your capabilities of running a website on your own. If you’re running MultiSite and you can’t do it, you’re going to be in a world of hurt one day, and I will do the ‘Told you so’ dance.


  • The dangers of an unchecked MultiSite?

    Blogetery was shut down, mysteriously, over the weekend. It was a WP MultiSite setup, with around 70k blogs. Not terribly abnormal to have an install that big, but the thing as an unnamed law enforcement agency shut them down. Details, such as they were, were posted at ReadWriteWeb: 70,000 Blogs Shut Down by U.S. Law Enforcement. Their shutdown reminded me of the hazards of running a website where anyone can register and make their own site and how important it is to be vigilant about what shows up on your website.

    Discussion of the situation spun up on Web Hosting Talk where it was determined that Blogetrey had been accused of hosting inappropriate content before. That probably meant they were hosting torrents or other illegal but not shut-down worthy. Copyright infringement. The site owner claimed that every copyright violation was removed within 24 hours. By the way, if you ever get slapped with a DMCA notice (i.e. a notice that your site has content copyritten to someone else), in order to be safe from a law suit, all you have to do is remove it. Done.

    So what on earth would cause BurstNET, their host, to shut down the site without warning or notice? That’s right, he had to ask ‘What happened to my site?’ and was told it was shut down, terminated, and here’s his money back.

    Turns out he had a link.

    From BurstNET’s statement:

    “It was revealed that a link to terrorist material, including bomb-making instructions and an al-Qaeda “hit list”, had been posted to the site. “

    That’s it. A link. One link. But it was enough for a warrant which then showed this:

    “Upon review, BurstNET® determined that the posted material, in addition to potentially inciting dangerous activities, specifically violated the BurstNET® Acceptable Use Policy. This policy strictly prohibits the posting of “terrorist propaganda, racist material, or bomb/weapon instructions”. Due to this violation and the fact that the site had a history of previous abuse, BurstNET® elected to immediately disable the system.”

    Now the previous ‘abuse’ was copyvio, which was all handled legally, but clearly BurstNET was feeling the pinch. They probably got slapped with a wwarrent and did the legal thing: They shut it down.

    Reagrdless of if it was fair or not to the other 69,999 sites hosted by Blogetery, it brings up the inherent problems of running an unchecked MultiSite. Anyone can make a blog/site, anyone can update it, and anyone can get you in trouble.

    It’s been a few weeks, but finally news is coming out about the whole story. CNET’s article was invectively titled Bomb-making tips, hit list behind Blogetery closure. That said, it explained this in more detail which let everyone get a grip on what was actually going on.

    I’m not going to get into the ethics of free speech and how it does (and doesn’t) apply to your website. Instead I want to use this as a reminder of the trouble you can get into, hosting websites. I host four, three are ‘mine’ and one is a site I like and visit pretty often. I’m very much aware of what’s going on all these sites and I monitor them frequently. This is not just to my benefit, but to everyone else’s on my servers. My host would be 100% within their rights to say “Ipstenu’s got a site that has kiddie porn! Kill her account!” and that would shut down everyone on my server.

    As I mentioned before, WordPress MultiSite makes it a lot easier for someone to host a thousand blogs, unchecked, but that also means it’s a lot easier for someone to post questionable content. For copyvio cases, you’re covered when you remove the material in question, but for porn and terrorism, it’s not actually under the same purview. Again. I’m NOT going to get into the why of this, nor the right or wrong about it. If you have a website, you have to accept that your host really has no interest in being involved with a legal dispute regarding kiddie porn or terrorism.

    This means it’s down to you to constantly and consistantly monitor your site for sub-sites and domains that are questionable. For me, if a site I host gets one Cease and Desist about copyvio, I take down the material, explain to the person who runs the site why, and ask them not to do it again. At this point, it’s their job to monitor their site. Should they fail to do so a second time, I give them a final warning of ‘If you can’t keep tabs on your site and your visitors, you can’t stay here.’ Third time and I close their account, refund them what’s left on their time, and offer to give them a copy of their site and database, intact.

    For the rest, though, it’s a no-warning termination, specifically because porn and terrorism are hot button topics. I’m within my rights to do so (I own the server, I make the rules) and I owe it to the other people. My ISP is in their rights to do similar, because they own the … land my server is on. If that makes sense.

    If all this sounds like too much work for you, then you shouldn’t be running an open, anyone-can-register-and-blog, multisite. Or you should hire some staff. Multisite is not a quick money scheme, it’s a job, and you have to take it seriously.

    This is not endemic solely of WordPress, but with the advent of MultiSite becoming mainstream, it’s something that’s going to start coming up more and more. Don’t say you weren’t warned.

  • The WebSVN Alternative

    The WebSVN Alternative

    Now that SubVersion is up and running, I decided to make it visible to others via WebSVN. The only reason I’m using WebSVN, which is really cool, don’t get me wrong, is because I can’t get mod_dav_svn and the default apache stuff working with my server.

    Go to http://svn.ipstenu.org and you can see my web-version of SubVersion up for you to snag whatever you want.

    Installing and configuring this was really easy. I followed the directions. Done. The only tweak was that I had to finagle some Open Basedir settings to let my install talk to SubVersion. But it works, and now you can see what I’m working on.

  • Struggling with SubVersion

    After some twittering with Mark Waterous, we decided that I probably had an out of date version of SVN. This is not a surprise. My server is CentOS, and they have this weird ‘core’ plugin repository that basically says ‘whatever version of a package we release your OS with is the version you get’. It took some heavy lifting and a stern conversation with yum that I do too want to use rpmforge as a source, but I finally upgraded everything to version 1.4.6 (previously I was on 1.1.x).

    And what happened when I tried to reactivate mod_dav_svn

    Same damn error!
    API module structure 'dav_svn_module' in file /usr/lib/httpd/modules/mod_dav_svn.so is garbled - expected signature 41503232 but saw 41503230 - perhaps this is not an Apache module DSO, or was compiled for a different Apache version?

    I took a step back and decided to make sure SubVersion really was working on my server. After a couple hours of wrangling, I determined it was. In fact, it’s working fine. For a while I had svnserve running and you could see it up on a website. But it still wouldn’t let me check in or out files from my PC at work (I had to do it all via command line on the server). That’s not what I want.

    So why won’t dav_svn work? No idea. Mark implied it’s CentOS and I’m inclined to agree. Everything else works. Not that I really have a thing against CentOS, but Red Hat (and it is Red Hat) has been weird ever since they released their IPO a decade back. Lost their street cred, I suppose.

    Mark’s suggestion:

    Pain in the butt, but uninstall current, try mod_dav_svn-1.4.6-0.1.e14.rf.i1386, then 1.3.2, then 1.2.1, see if any work?

    Thankfully, installing older packages in yum is pretty easy. It’s just yum install mod_dav_svn-1.4.6-0.1.el4.rf.i386 and done. Except I got this:

    ---> Package mod_dav_svn.i386 0:1.4.6-0.1.el4.rf set to be updated
    --> Running transaction check
    --> Processing Dependency: subversion = 1.4.6-0.1.el4.rf for package: mod_dav_svn
    --> Finished Dependency Resolution
    Error: Missing Dependency: subversion = 1.4.6-0.1.el4.rf is needed by package mod_dav_svn

    Basically, subversion and mod_dav_svn have to match for this to work. Now, since I have subversion working, I really don’t want to roll that back. I decided that maybe I just had a bad copy from the repository, so I downloaded SubVersion and installed it manually. This is always fun, but I do it often enough at work that I’m only slightly terrified. This is why I have a managed server, by the way. Since I got the same damn error after a manual install, I decided that something was wrong with my server and mod_dav_svn.

    Thankfully there are alternatives!

    svnserve is not as ‘pretty’ but it works. I followed the default installation directions and then used Subversion : svnserve over xinetd to get it running. Once I got it running, I was able to point things at svn+ssh://ipstenu.org/svn/repos/ipstenu (which will do you no good, it’s locked to my user ID/password). Now everything downloads and I say sayonara to Google Code!

    Next up, I’ll put a WebSVN site up for people to browse my code. If they really want to.

  • SubVersion Woes

    SubVersion Woes

    Since my webhost, the utterly wicked cool Liquidweb, already has SubVersion installed, I thought ‘Wouldn’t it be nice to get OFF of Google Code and onto my own site.’ Step one was turning my site into a MultiSite install, but that’s actually another story. Step two, which is this step. See, I have SubVersion (here after referred to as SVN) on the server. What I didn’t have was a way to go to http://svn.ipstenu.org and get things like my plugins.

    I’m currently having all sorts of shenanigans when I tried to get this up and running. There’s no one good doc on how to do it all, and while I tried to make this into one, I believe the term ‘Epic Fail’ comes to mind.

    Following the CentOS directions, my install fails when I try to add these lines
    LoadModule dav_svn_module /usr/lib/httpd/modules/mod_dav_svn.so
    LoadModule authz_svn_module /usr/lib/httpd/modules/mod_authz_svn.so

    to my httpd.conf, I get a fun error.

    httpd: Syntax error on line 28 of /usr/local/apache/conf/httpd.conf: Syntax error on line 1 of /usr/local/apache/conf/includes/pre_main_global.conf: API module structure 'dav_svn_module' in file /usr/lib/httpd/modules/mod_dav_svn.so is garbled - expected signature 41503232 but saw 41503230 - perhaps this is not an Apache module DSO, or was compiled for a different Apache version?

    When I look at it, I see I’ve got this version:
    # yum info mod_dav_svn
    Setting up repositories
    Reading repository metadata in from local files
    Excluding Packages in global exclude list
    Finished
    Installed Packages
    Name : mod_dav_svn
    Arch : i386
    Version: 1.1.4
    Release: 3.el4_8.2
    Size : 96 k
    Repo : installed
    Summary: Apache server module for Subversion server.

    Description:
    The mod_dav_svn package allows access to a Subversion repository using HTTP, via the Apache httpd server.

    At this point, I think I have a bum version of mod_dav_svn. Or I’m benig too dumb for SVN.

  • I take it back. WP-Super-Cache is a Super Hero

    cupspikes I’m going to be upfront and admit that I’ve never actually liked this plugin. A very large part of me wants to side with Matt Mullenweg in that if you have a good server, configured properly, with a decent host, you should be just fine. Also, it doesn’t really work well with my favorite anti-spam plugin, Bad Behavior, which stops 99.999% of my spam cold. But. Over the years of running a vaguely popular fan site, I’ve been nailed by service spikes that killed me and everyone else on my shared hosting setup (multiple websites, not all connected, sharing a virtual server). At one point, I had to offload ‘news’ to LiveJournal, but since then, I’ve pulled it all back to WordPress, moved to a virtual private server (VPS, just me and my sites on a virtual server) due to the need for better support, and I was kind of complacent. Things were trucking along just fine, we had some major news that were handled without a blip, and I thought I was cool.

    Yesterday I had to cycle the HTTP service three times to clear things up. The first time, someone was using a really old URL (for a part of the site gone 2 years now) and, when it didn’t give them what they wanted, they kept hitting it. I blocked the IP address and we were fine. But. Then the news that I had some new, cool, information hit, and suddenly I was spiking like mad. I checked my stats, trying to see what was the culprit. The gallery is pretty tolerant of these things (though I have turned on the Static HTML cache right now) and while I did have some hefty images (1 to 2 MB, I usually try to keep ’em to .75 megs), it wasn’t ZenPhoto borking.

    No, my poor, poor WordPress was having a heart attack because I’d gotten myself crosslinked from a couple high traffic sites. How bad? Well that spike on the graph below may explain it.

    spiked

    The first thing I did was tune the server. Actually, I’d done that months ago, dropping my memory usage from 77% to about 60%, but now I went in to see how well that was working. There was a little more I could do, so I optimized a couple more settings and things eased up a little. Not enough. I scrubbed the CPU usage, too, and normally we never spiked over 1 for a load average, but that wasn’t working yesterday. Sidebar. CPU Load is a very bizarre thing to most newbie server admins, and I’m not great at sorting it out myself. Of course, I know that a ‘good’ load is anything around .3 and a ‘bad’ load is something like, oh, 9. And yes, I hit 9 yesterday on my 8 processor VPS box. I’m not going to explain it here, as I’m still learning and I’m sure I’d get it wrong, but the gist is that you don’t really worry if your load hops up to 1 or 2 for a short amount of time. When it stays there and is spiking at 4 or 5, however, you need to pay attention.

    What kept happening to me was that it would spike up to a load between 5 and 9, and the HTTP service (the bit that serves up webpages) would scream and fall over. Email, FTP, shell access and the rest were all okay, though, so I knew the server itself was fine. Thus I deduced something was sucking up the load and I knew I had three choices: JFO’s blog (it’d happened before), JFO’s gallery, or YTDaW. While I host YTDaW, I don’t actively admin it in any authoritative stance. The only ‘mod’ work I’ve done is turn off email alerts for people who are using non-existent emails (and then, only when I’m tired of getting their bouncing email). Devon pretty much keeps me on tap for server admin and security stuff, and I do my utmost best to keep my hands OUT of the pie. It’s her baby, I’m just the tech.

    And while they’re using a pretty old version of the forum software, it’s secure enough and solid enough that I didn’t think they were the culprit. The evidence (heh) supported that theory, so I went to look at JFO. It was definitely my old girl, and right away I could see that we were getting a lot of traffic from new users. Four times the traffic. Before you could say ZOMG! I was on Google Analytic and Woopra, checking out who the hell was hitting my site and the answer was surprising.

    Everyone. (Well, mostly FaceBook, AfterEllen and Twitter, but really, it was all over.)

    I’d accidentally broken news about three hot topics within a couple hours, and now everyone and their mother wanted to see JFO and, as many people have mentioned, WordPress was hemorrhaging under the ‘digg’ effect. Basically it was trying to serve up dynamic (generated on the fly) pages to too many people at once. If I was using static HTML, it would go faster, but WordPress doesn’t do that. Except … except it does if you use WP Super Cache.

    no10 As I mentioned before, I don’t (didn’t!) like that plugin. I want my app to behave correctly without it. I mean, the PM of Britain uses WordPress! I was sure they don’t need caching. They probably have a rack of servers on a co-located cluster. Except I viewed source and they were using it. The Library of Congress wasn’t, though, and neither were The Speaker of the House (Nancy Pelosi) or the Army. Honestly, I wasn’t sure how to take that, but after four hours of babysitting my server, I took a plunge and installed WP Super Cache for the fourth time.

    The first few times sucked, I admit. It was a lot of massaging and manual crap that, while I’m perfectly capable of doing, I didn’t like. This was easier. A chmod, an install, a click, another chmod and then I was done. And guess what? My loads dropped from an average of 3.45 to one of .35 by morning. On top of that, my memory had one spike since I turned it on, and that was right when I was running backups and the like.

    memoryspike

    So I’m keeping it on for now, especially with what I expect tonight, but I think that I can say … yeah. WP-Super-Cache does what it says.