Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: wordpress

  • Hack’n’Slash Security

    I was intending on a totally different post, but, well, this came up instead.

    Recently, WordPress, my preferred blogging software, has been under attack by both hackers and critics. There were actually three attcks that all got lumped into one so I’ll try and break this down. If you’re of the ‘Too long! Didn’t Read!’ variety today, you can get by with knowing this: If your WordPress install is not secure and if your web host is not secure and if YOU do not follow security practices, then you will be hacked. Period. Security relies on you, your web host and your web apps all being sensible about the whole thing to be effective. Remember, it’s okay to ask for help!

    Also go read Hardening WordPress right now.

    Okay, so security.

    Back in Feburary/March, there was a sudden influx of users complaining their sites had been hacked by inii.info, whereby the hack was to edit the wp-blog-header.php and change it so any time a search engine bots visited your site, they went to inii instead. This matters because search engine bots collect information about your site and use it to rank your website against all the other sites about a given topic. There was a second hack where a file named ... (yes, three periods) had even more redirect code in it. And it was heavily encoded so you couldn’t read it without decoding.

    The reason I call this a Media Temple hack was that it seemed to be prevalent to Media Temple installs. While at first people jumped the gun and said ‘It’s WordPress!’ Media Temple came out with a detailed Q&A about the matter and the attack appeared to affect ALL webapps via compromised passwords. If Media Temple ever revealed what happened, I’m not aware of it, but it wasn’t just WordPress that was affected. They ended up changing DB passwords for every webapp, from Drupal to vBulletin.

    In early April, there was another rash of hacks, this time targeting Network Solutions. This time, it looked like a clear cut case of database changes. WordPress, like most PHP/SQL apps out there, uses a database to store all its information. In this instance, the database entry for the site’s URL was changed from (for example) https://ipstenu.org to an iframe link I’m not reproducing here.

    At the same time, there was a ‘Pharma’ hack, where links with ‘pharma’ in them were slipped into your site, in a rather genius fashion. Chris Pearson has a decent explanation on the matter, but I feel he’s barking at the wrong car for part of it.

    Chris and Media Temple and Network Solutions and a horde of people on Twitter and forums every where jumped up and said “AHA! It’s a WordPress hack!!!111!” Which … well, yes, but not exactly. As the very wise Andrea_r put it, there’s a difference between attacking WordPress installs and targeting WordPress installs.

    An analogy if you please. There’s a rash of break-ins in a small town. The houses that are broken into are all bungalows. People shout ‘Aha! It’s a problem with bungalows not being secure!’ The police look into the matter and find out that in every house broken into, the bathroom window was left open. Now, is this the fault of the builder, who designed bungalows to have a window people could fit in through or is this the fault of the residents who didn’t close and lock their windows?

    If you said ‘It’s a little of each!’ then thank you, you can stay after class and clean the erasers.

    Security depends on many things, but to the topic at hand, server security is a tripod, and relies primarily on these three legs:

    • The Web Host is responsible for making sure the sever itself is up to date with the latest patches etc, and that the server is configured in a safe way.
    • Web-apps are responsible for not unleashing needless insecurities to the system.
    • The end-user we pray to the flying spaghetti monster that they’ve not done something to violate security out of ignorance.

    To understand how these hacks all worked, yes all of them, you have to look at the perfect storm. This is what had to happen in order for all these accounts to be compromised:

    1. Someone saved their wp-config.php file in a way that it was readable by the free world.
    2. Someone scanned for and found that file.
    3. The user was using their ID and Password, rather than creating a DB user just for the blog.
    4. That account had read access to other accounts on the same server
    5. The malicious user used the account to scan for other wp-config.php files, even if they were saved securely and compromised their accounts/databases as well.

    That’s a lot of wrong on one box. With most webhosts, you’re on what’s called ‘Shared Hosting’ which means a whole mess of people are on the same server, each with their own ID and password. Much like if multiple people have IDs on a desktop PC, the inherent security of the server does not allow Joe to look at Jane’s files, unless she saves them in a public space. Alas, one a couple sites, this was not the case. SO Joe, who saved his wp-config.php file with 777, and used his server ID and password to access his database in that file, was compromised. And once the hacker had Joe’s information, he scanned the entire server and hurt everyone.

    Ouch.

    But wait, doesn’t that mean it’s WordPress’ fault for saving passwords in the wp-config.php file in a way a hacker can read them!? Well, yes, it’s certainly WordPress’ ‘fault’ but you have to realize that doing so is an accepted risk of most PHP/SQL webapps, in that for the SQL DB to be read, the password to that database must be kept in clear text (i.e. not encrypted). This is in the wp-config.php file.

    Okay, so it’s Joe’s fault for saving his file in a readable fashion? Somewhat. By having their wp-config.php file set so that anyone can read it (bad permissions – 777 for example), Joe put himself at risk. This IS NOT a flaw in web-app or the ISP, it’s just … well, ignorant (unless the ISP is forcing the file to be 777 to run WordPress, at which point it’s their fault, and yes, there’s an ISP that does that!). In addition, I know a lot of people who, instead of making a DB user for their blog, will put their server ID and password in that file, which means once it’s been read, ANYONE can log into that server as them. I suspect this is done from ignorance as well. By the way, your server ID and password is the same as your FTP user ID and password in most cases.

    Back to WordPress, shouldn’t they check for that? Maybe. But it’s not that easy, since there are a lot of different ‘acceptable’ security settings for that file, and it all depends on the server. Maybe one day WordPress will figure that out, but right now they tell you to make it secure.

    What about the web server? They are responsible for making sure that if Joe User set his WP config file to 777, and put their server ID/Password in there, the worst they can do is shoot themselves in the foot by preventing them from reading anyone else’s user directory. Limit the destruction on a per-user basis. There are a lot of Shared Hosts out there with lax security policies, which makes this more prevalent than I’d like.

    Hopefully that made sense.

    All of these hacks seem to be looking for people with wp-config files that can be read, logging into the account as the user (or the database user), and either adding files that edit the database, editing the database, or both editing the database and adding the fake plugin files.

    Once your server is insecure, because of compromised IDs and Passwords, you have to go back to zero, reset ALL your passwords, scan your PC for viruses, and be careful. Remember, if they have your password, they can do everything you can do.

    Good luck out there. Be smart, be secure, be safe.

    Edited to add…
    Also check out Mark’s well written post about how your security? Is your responsibility. Because dude, is SO is.

  • But If, Baby, I’m The Bottom, You’re The TOP

    Earlier this month I talked about how my server was acting wonky and how I fixed it using, among other tools, TOP.

    This week I was chatting with a fellow about CPU usage and his site. He runs a rather large WordPress blog and the database is about 500 megs. As a comparison, this site, with about 500 posts, is under 5 megs, and my big site, with thousands of posts, comments, and a forum, is 10 megs. The biggest site I run on my server is 850 megs (just down from 910 after some clean up). The difference between his site and mine is that his is slow and he knows it. As we discussed ways to speed it up, I had some thoughts on WordPress and how, at a certain point, you’re going to need to dig into the guts of your server and learn TOP.

    The ‘problem’ with most ‘How do I make my WordPress site run faster?’ tutorials, as I’ve seen it, is they address surviving the digg effect. That is, they talk about how to deal with having a high volume of traffic on your site and, for the most part, you can make it with just adding caching plugins.

    Once your site gets ‘big’ or ‘popular’ you’re going to have to move off shared/cloud hosting and over to your own server. For most of us, the first step is a VPS (Virtual Private Server). Shared Hosting means ‘You have an account on a server with a hundred other people.’ It’s great for small sites, inexpensive and easy to use. The problem is you could have terrible neighbors, who use up all the CPU. Think of it like those old New York apartments where someone’s a jerk at 5am and uses up the hot water so you, at 7am, have none. Yeah, it’s kind of like that. That’s the day you think ‘I want a house!’

    Only, well, we’re not all up for houses just yet. A house would be a dedicated server, where it’s just you. Cloud hosting, which I touched on earlier, would be the college dorms of webhosting. It has a lot of benefits for the really small sites, and actually some for large sites, but I’m not sold of their overall usefullness yet, so I’ll talk about them some other time. What I want to talk about are Virtual Private Servers, the condo-sub-leasing (or rent-to-own maybe) of website hosting, and how the new VPS user should really get on TOP of things (sorry, bad pun) to make their lives easier.

    TOP. Well ‘top’ really. Unix commands are generally all lower case like that.

    The top command is a system monitor tool that outputs a list of processes. Have you ever seen Task Manager in Windows? It’s kind of like that tab for ‘Processes’ that you look at and run away from. The default view of top is by percentage of CPU usage and the “top” CPU users are listed. See? The name made sense. You can also see how much processing power is being used, memory hogs and other cool things. Most modern Unix-systems let you sort the list, colorize it, etc, though you have to be command line savvy.

    Here’s what top looked like for me about an hour ago.

    top - 12:44:44 up 126 days, 23:13,  1 user,  load average: 0.12, 0.17, 0.17
Tasks:  91 total,   1 running,  90 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    524288k total,   358248k used,   166040k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15616 nobody    15   0 94540  65m  20m S  1.0 12.8   0:00.27 httpd
12261 ipstenu   16   0  1908 1012  780 R  0.4  0.2   0:00.30 top
[...]
28630 root      16   0  107m  86m 1096 S  0.0 16.8   1:06.30 /usr/sbin/clamd

    I wanted to point out clamd, which has been the bane of my existance. Thing won’t DIE. I ended up going in to /etc/exim.conf and manually commended out the clamd line (and restarted the service) to finally get it gone.

    But top, as you can see, has a freakishly large amount of information. My server is doing fine, at this point, so I don’t have a whole lot to show you. What you can see right away is that I can tell, with a glance, what’s going on. I could see, though and at this point I have a ‘nobody’ process. That just means someone’s accessing my website. No, really! That’s good! The CPU and memory usage seem high, but they vanish in a second. Basically, someone rang my doorbell and for that brief moment, electricity was used. The next thing I see is the top command, which is run by me (hi!) and down the line is that idiot, clamd.

    I actually scan top a lot at work these days, trying to understand what’s causing issues. It’s good for ‘right now!’ things, but not so much if I want to see what started a strange spike a couple hours (weeks) ago. For that you need a whole mess of tools.

  • I take it back. WP-Super-Cache is a Super Hero

    cupspikes I’m going to be upfront and admit that I’ve never actually liked this plugin. A very large part of me wants to side with Matt Mullenweg in that if you have a good server, configured properly, with a decent host, you should be just fine. Also, it doesn’t really work well with my favorite anti-spam plugin, Bad Behavior, which stops 99.999% of my spam cold. But. Over the years of running a vaguely popular fan site, I’ve been nailed by service spikes that killed me and everyone else on my shared hosting setup (multiple websites, not all connected, sharing a virtual server). At one point, I had to offload ‘news’ to LiveJournal, but since then, I’ve pulled it all back to WordPress, moved to a virtual private server (VPS, just me and my sites on a virtual server) due to the need for better support, and I was kind of complacent. Things were trucking along just fine, we had some major news that were handled without a blip, and I thought I was cool.

    Yesterday I had to cycle the HTTP service three times to clear things up. The first time, someone was using a really old URL (for a part of the site gone 2 years now) and, when it didn’t give them what they wanted, they kept hitting it. I blocked the IP address and we were fine. But. Then the news that I had some new, cool, information hit, and suddenly I was spiking like mad. I checked my stats, trying to see what was the culprit. The gallery is pretty tolerant of these things (though I have turned on the Static HTML cache right now) and while I did have some hefty images (1 to 2 MB, I usually try to keep ’em to .75 megs), it wasn’t ZenPhoto borking.

    No, my poor, poor WordPress was having a heart attack because I’d gotten myself crosslinked from a couple high traffic sites. How bad? Well that spike on the graph below may explain it.

    spiked

    The first thing I did was tune the server. Actually, I’d done that months ago, dropping my memory usage from 77% to about 60%, but now I went in to see how well that was working. There was a little more I could do, so I optimized a couple more settings and things eased up a little. Not enough. I scrubbed the CPU usage, too, and normally we never spiked over 1 for a load average, but that wasn’t working yesterday. Sidebar. CPU Load is a very bizarre thing to most newbie server admins, and I’m not great at sorting it out myself. Of course, I know that a ‘good’ load is anything around .3 and a ‘bad’ load is something like, oh, 9. And yes, I hit 9 yesterday on my 8 processor VPS box. I’m not going to explain it here, as I’m still learning and I’m sure I’d get it wrong, but the gist is that you don’t really worry if your load hops up to 1 or 2 for a short amount of time. When it stays there and is spiking at 4 or 5, however, you need to pay attention.

    What kept happening to me was that it would spike up to a load between 5 and 9, and the HTTP service (the bit that serves up webpages) would scream and fall over. Email, FTP, shell access and the rest were all okay, though, so I knew the server itself was fine. Thus I deduced something was sucking up the load and I knew I had three choices: JFO’s blog (it’d happened before), JFO’s gallery, or YTDaW. While I host YTDaW, I don’t actively admin it in any authoritative stance. The only ‘mod’ work I’ve done is turn off email alerts for people who are using non-existent emails (and then, only when I’m tired of getting their bouncing email). Devon pretty much keeps me on tap for server admin and security stuff, and I do my utmost best to keep my hands OUT of the pie. It’s her baby, I’m just the tech.

    And while they’re using a pretty old version of the forum software, it’s secure enough and solid enough that I didn’t think they were the culprit. The evidence (heh) supported that theory, so I went to look at JFO. It was definitely my old girl, and right away I could see that we were getting a lot of traffic from new users. Four times the traffic. Before you could say ZOMG! I was on Google Analytic and Woopra, checking out who the hell was hitting my site and the answer was surprising.

    Everyone. (Well, mostly FaceBook, AfterEllen and Twitter, but really, it was all over.)

    I’d accidentally broken news about three hot topics within a couple hours, and now everyone and their mother wanted to see JFO and, as many people have mentioned, WordPress was hemorrhaging under the ‘digg’ effect. Basically it was trying to serve up dynamic (generated on the fly) pages to too many people at once. If I was using static HTML, it would go faster, but WordPress doesn’t do that. Except … except it does if you use WP Super Cache.

    no10 As I mentioned before, I don’t (didn’t!) like that plugin. I want my app to behave correctly without it. I mean, the PM of Britain uses WordPress! I was sure they don’t need caching. They probably have a rack of servers on a co-located cluster. Except I viewed source and they were using it. The Library of Congress wasn’t, though, and neither were The Speaker of the House (Nancy Pelosi) or the Army. Honestly, I wasn’t sure how to take that, but after four hours of babysitting my server, I took a plunge and installed WP Super Cache for the fourth time.

    The first few times sucked, I admit. It was a lot of massaging and manual crap that, while I’m perfectly capable of doing, I didn’t like. This was easier. A chmod, an install, a click, another chmod and then I was done. And guess what? My loads dropped from an average of 3.45 to one of .35 by morning. On top of that, my memory had one spike since I turned it on, and that was right when I was running backups and the like.

    memoryspike

    So I’m keeping it on for now, especially with what I expect tonight, but I think that I can say … yeah. WP-Super-Cache does what it says.

  • Plug It In, Plug It In

    vilcus-plug-it-inI am not a great programmer by any means. I can hack around and muddle my way through with the best of the great net scapegraces. I’m not the genius who invents a brand new way of doing things. That said, I do, eventually, get annoyed with things enough that I force myself to learn how to code.

    Yesterday I was pissed off at WordPress because of it’s user management tools, and no plugins really did what I wanted. See, I have open registration. It lets me sync my blog and forum and let people post. But where it fails is that I can’t set users as ‘banned’ in WordPress. This is a simple thing, I feel. A user role that has no rights and is just banned from commenting. They can read all they want, but no comment. I’ve tried just about every tool out there, but they never work. In addition to that, spammers sign up to my blog.

    Since creating a ‘bozo’ user role is outside my ability, I decided what I wanted was a plugin to prevent people from registering if they were on my blacklist, similar to how I can prevent them from commenting on my comment blacklist. At first I was using TimesToCome Stop Bot Registration, which (among other things) uses StopForumSpam’s list of spammers as a stop-gap.

    The problem with TTC is that if you register with a bad email (jane132@gmail.com instead of jane123@gmail.com) and then try to register with the RIGHT email, it notes that the IP is the same and bans both emails and the IP. Which caused a couple people no end of problems on my site. It had to go.

    From there, I tried No Disposable Email, which checks against a list of known baddies. That was nice, but it was a text file list that you had to update by hand. But it got me thinking.

    I quickly converted it into Ban Hammer, which allowed me to update and edit the text file from a submenu inside my admin session. But that wasn’t enough. Why did I have to have two places to keep my jerk list? If someone was on my WordPress Comment Blacklist, I didn’t want them to comment. That implies they’re just not welcome at all. So why don’t I make Ban Hammer pull from that list. Which I did.

    I still have things I want to do to the code, like put in an option to use StopForumSpam’s list, and a way to edit the error message. But for now, Ban Hammer sits by my other plugins, Recently Registered (lists the last 25 registrations) and my bbPress plugin Spoiler Bar (adds in spoiler ‘code’ to bbPress) on my Google Code site. It’s not for ‘public’ release, but it’s there so my friends who have been helping me test out my ideas can easily download. What? I have nerd friends!