Tag: administration

All Subs Are Equal

For years we’ve told people “The only difference between subdomains and subfolders is that search engines see subfolders as being all a part of the same site, and subdomains as separate sites.

That may not be true for very much longer.

As August ended, Google dropped this little bomb casually:

Most people think of example.com and www.example.com as the same site these days, so we’re changing it such that now, if you add either example.com or www.example.com as a site, links from both the www and non-www versions of the domain will be categorized as internal links. We’ve also extended this idea to include other subdomains, since many people who own a domain also own its subdomains—so links from cats.example.com or pets.example.com will also be categorized as internal links for www.example.com.

via Official Google Webmaster Central Blog: Reorganizing internal vs. external backlinks.

This is, I think, a good thing. They make sure you understand that ipstenu.wordpress.com won’t be the same as jane.wordpress.com, which makes sense since their own blogger platform runs off subdomains as well. Somewhere in their logic they know ‘Aha! Ipstenu doesn’t own wordpress.com! But she does own Ipstenu.org.’

To reiterate, this only affects things in Google’s Webmaster tools:

This update only changes how links are displayed in Webmaster Tools. It doesn’t affect how links are valued in relation to the search algorithm or ranking. It has nothing to do with Panda, nothing to do with keywords, nothing to do with PageRank.

I fully expect that it’s going to change and expand. Since most of us end up registering our ‘master’ domain (i.e. ipstenu.org) off Google Webmaster, they can easily leverage the data they already have in order to tweak search engine results. Another tactic would be to start using a new meta type tags. After all, the big guys already organized schema.org (which isn’t as picked up as it might be yet).

Sidebar: Schema.org’s problem is how complicated it is to fold into CMS tools. If they’d started by releasing this with a couple plugins/extensions for the popular open source tools like Drupal, Joomla, MediaWiki, MovableType and WordPress, which as they’re open source, they could have, we’d be way ahead of the game. As it stands, the one WordPress plugin I’ve seen requires you to import SQL tables! If they get schema ironed out so it’s simple and easy to use, then we can add in ‘parent’ or ‘main’ site parameters to our code and every search engine can know ‘halfelf.org is internal to ipstenu.org, but taffys.ipstenu.org is not.’ And wouldn’t that be cool!

Personally, I have ipstenu.org, ipstenu.org, halfelf.org and so on all listed separately in Google Webmaster right now. I could do that with subfolders as well, to track data internally to itself (i.e. each subdomain/subfolder), and still track them at the master level with the main domain.

So no effect on your SEO or ranking or any of that stuff, but a nice change for Webmaster Tools.

Where do you see this going?

19 September, 2011
TimThumb and the Pseudo (D)DoS Effect
Over the course of a day, my server rebooted httpd twice. That’s not a common thing for me, as after days and hours of work, I managed to finegal my server into a semblance of stability so it could handle massive loads. At first I thought it was due to my traffic spike of an increase of about 1500% (no I did not screw up a decimal place there, my traffic went from a couple hundred to nearly 4000 in one day). Then I thought that couldn’t be right, because my traffic had actually mellowed out by the time I was crashing.

So I went to my emails and pulled up the Apache Status log and noticed that 70% of the calls to my site were GETs to pages like this:

/wp-content/themes/newsworld-1.0.0/thumb.php?src=/g0../0d1. /wp-content/themes/DeepFocus/tools/timthumb.php?src=/g0../0 /wp-content/themes/TheProfessional/tools/timthumb.php?src=/

And thanks to that massive spike in traffic, my server was slowing down to the point that HTTP was becoming unresponsive and it had to reboot itself. In short, the TimThumb exploit was causing my server to behave like it was under a Denial of Service Attack, even though I don’t use TimThumb! My server was able to handle this, but if I’d been back on my old Shared Server, I’d probably have not gotten a text from the server at 11pm saying “Man, we had to reboot, but it’s okay now. Can I have a beer?”, but instead woken up to ‘Dude, where’s my website!?’ And this is with having a fantastic web host who cares and takes the time to help me out.

Normally this is where I’d tell you what to do if you’ve been infected via the TimThumb exploit, but Cleaning Up the TimThumb Hack covered it pretty well. Just remember this, if you have been infected, you must reset all your passwords. This is true of any and all hacks. As soon as someone has access to meddle with files on your server, you could be hurt worse than you know. At the very least, you need to read the post “Technical details and scripts of the WordPress Timthumb.php hack” by the guy who ‘fixed’ TimThumb.

What I wanted to do here was sort out how to block people who were looking for timthumb.php files (I can’t block thumb.php as I use that filename elsewhere). Reading up on Perishable Press’s Stupid .htaccess Tricks it’s clear we can do this:
```
# prevent viewing of a specific file

 order allow,deny
 deny from all
```
That should simply block access. An .htaccess block is a pretty simple way to reduce your server load, because people are getting punted before they get very far into things. Still, it’s something I have to install on each account on my server. Right now they’re just hammering ipstenu.org, and this is not the only domain on my server. This is, by the way, the same problem with using a plugin like WordPress Firewall. It’s a fantastic idea, if all you have is one account on a server. Protect yourself.

I don’t. I run a VPS, and I have four domains here which I should be protecting. It’s easy enough to make that change on all four, plus two other external servers, but is that the best use of my time? I say no. I think I should automate as much of this as I can. What I really want is to say ‘If you’re looking for timthumb.php, then the odds are you’re looking for security vulnerabilities, and you need to just die in a fire.’ Or at least firewall block them. Seeing as I already have CSF on my server, it was logical for me to start there.

Blocking an IP is easy, and I can do it via command line or GUI. Auto-detecting a URL though, is more complicated. Part of me thinks that, much like I can auto-block someone who tries to log in with any ID 10 times in a row, I should be able to add in a file pattern somewhere. Turns out you can’t, at least not the way I wanted. Instead, you have to do it differently.

TimThumb’s exploit scanner isn’t actually a DDoS Attack, but it acts like one. A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make your site unavailable. Basically they hit your site with so much traffic that it’s incapable of its job: serving up webpages to happy readers! That’s why I call this a pseudo (D)DoS attack. The goal of the scanner is to see if you’re using Timthumb and, if so, to be evil. It’s not really distributed (i.e. multiple computers at once), though because of the number of people running the exploit scanner, it can seem that way. The side effect is that your site is hammered and can’t do what it wants. Which leads us to Connection Tracking.

CSF has a tool called ‘Connection Tracking’ which lets you limit how many times a specific IP can hit your site at once before they get tossed to a shit-list. I set this to 300, and told it to only scan ports 80 and 443 (because I need to have unlimited FTP, and sometimes I log in from different IPs – yes, my home IP is on the whitelist).

Connection Tracking. This option enables tracking of all connections from IP addresses to the server. If the total number of connections is greater than this value then the offending IP address is blocked. This can be used to help# prevent some types of DOS attack.

Care should be taken with this option. It’s entirely possible that you will see false-positives. Some protocols can be connection hungry, e.g. FTP, IMAPD and HTTP so it could be quite easy to trigger, especially with a lot of closed connections in TIME_WAIT. However, for a server that is prone to DOS attacks this may be very useful. A reasonable setting for this option might be around 300.

Setting this up is a little less obvious for the new person. Go to WHM > Plugins > ConfigServer Security & Firewall and look for “Firewall Configuration”

Click on the button, look for CT_LIMIT and change it to 300.

Scroll down, click ‘Change’ and then restart CSF.

Now, you could put this as low as 100, or as high as you want, but I did some reading and 300 seems like something not too likely to trip innocent people up, but just enough to catch the bad guys. I may way to lower this to 200 or so, but I know that a lot of people come to one part of my server for the image gallery, and they tend to open lots of pages at once. I don’t want to hurt them. The other thing to keep in mind is how short is the block time. The IP block is set for 30 minutes, which isn’t much at all, but it could be just enough to make the transient DDoSers go away. ETA: As of February 2012, I’ve actually lowered this to 50, and it’s made a world of difference! My day to day CPU is a little higher, but the number of spikes that caused outages has dropped.

I’m not doing this to stop the people who want to bring my server to its knees. I’m doing it to stop the people who are ‘scanning’ for exploits. A true DDoS is hard to block because as soon as I block it, I have to defend against it again and again. CSF would be like a sump pump in your flooded basement, always running until it burns out the motor. It comes from too many sources, and for the little guy (i.e. me), I may just have to shut things down for an hour and wait it out. But these scanners, well, I can block them with this trick, and not hurt my server doing so!
6 September, 2011
United States Patent: 8010458

An online social network is provided in which members of the online social network control who may view their personal information and who may communicate with them. The members control who may view their personal information by setting a visibility preference. A member may not view another member’s full personal profile if the measure of relatedness between the two is greater than the visibility preference of the other member. The members also control who may communicate with them by setting a contactability preference. A member may not communicate with another member if the measure of relatedness between the two is greater than the contactability preference of the other member.

via United States Patent: 8010458.

It’s official. The US patent office is useless.

2 September, 2011
TimThumb, Heroism and FUD

FUD is “Fear, Uncertainty and Doubt” and it’s a tactic used by people to scare you and make you jump into a decision that benefits them. This decision may not be a bad decision, but it’s not strictly to your own benefit, but theirs. Keep that it mind, it matters.

Recently it was discovered that there was a massive vulnerability in TimThumb (TimThumb is an image editing tool for your webapps). It had an honest-to-god Zero Day Vulnerability. I don’t use the code, and I don’t put it on any site I run, so I knew I was pretty safe. Still, I ran searches for timthumb.php on my entire server, made sure it was clean, and moved on.(Not relevant, I recently changed all my passwords on all my sites, and my servers, because I realized I’d used the same ones for about 6 years.)

The exploit primarily affected WordPress installs, because it was developed for WordPress in the beginning, but since then has grown to be used by many other apps, like Drupal and Joomla and even home-grown ones. It’s insanely cool, but it’s always had weird little problems (which is probably why it’ll never be included in the core code of those apps). Getting it to work at all on MultiSite was a pain, and when someone wrote a how-two, we gave her Tim Thumbs up!(Bad joke. BAD bad bad joke. Sorry.)

Certain people leaped into action. VaultPress, which runs a backup service for WordPress users, sent out emails to everyone who had TimThumb. Then they went the extra mile and fixed 712 possible exploits for you. (I know some people got shirty about it, since they didn’t want VaultPress editing their data. That isn’t the point here.) They jumped up and said ‘We must fix things for people’ and did it. This was, indeed, Matt’s vision for VaultPress.

But then this other thing happened, and I’ll quote Matt:

It could have gone a lot of ways, but the incident brought out the best in the community. The core team sprang into action searching through the theme directory to inoculate any themes that contained the dangerous code. Community blogs quickly got the word out about the problem so people were aware of it. Mark Maunder, who originally discovered and broke down the problem, created a fork of the code called WordThumb that rewrote TimThumb from the ground up. Forking is not usually ideal because it fragments the market for users but Mark soon connected with Ben Gillbanks, long-time WordPress community member, and they’ve teamed forces to release TimThumb 2.0, a collaboration that exemplifies Open Source at its finest. An updated plugin should be in the directory shortly.

Let me explain. There was a problem with a popular tool that is used both in themes and on its own plugin (and probably others). Mark found the problem, fixed it, and then re-wrote the tool. Then, after Matt commented on his site that Forking is a last-resort, even though this was a ground-top rewrite, Mark agreed and talked to the TimThumb guy and together they fixed everything. And now they’re a team. No one made any money off that process. People just did the right thing to make the web safer for all of us! (Okay, that’s not the point either, but it needed to be made.)

All of this was done in a way that the public knew about the problem without getting into an “OMGWTFBBQ!!!11!?” panic. Was there some fear? Yes, because you knew there was a problem and there was a possibility it could affect you. Was there uncertainty? Of course! Again, could it affect you? Was there doubt? And this here is where we have a win for the Open Source community. There wasn’t. It was a straight up ‘This is what’s wrong, this is how to fix it.’

In so many ways, that’s how every business should work. It could have been better, certainly, but when I compare this to how I get security alerts for our servers at work, I see nothing but room for improvement. Right now, one person has the job to look for vulnerabilities that are published about anything we use. If she sees one, she opens a ticket and says ‘Fix this ASAP!’ The problem is, to use a recent example, ownership of the fix. We had a vulnerability with .NET, but as I read the whole doc, I sorted out that it only happened if your server was configured in a certain way so as to make a security hole. Another quick check and I saw the server team had their own ticket ‘Fix this hole.’ So I closed my ticket and said ‘Will be resolved with Server Ticket 1234567.’ My ticket was reopened and I was told I did it wrong. This was a problem with my application (I don’t ‘own’ .NET, I happen to be on the team who brought it into the company, however). I pointed out it wasn’t that .NET was vulnerable, that it was the server. They didn’t care. My ticket has to be open until the problem is resolved, no matter what. In the end, I turned off the feature that might, possibly, be vulnerable and got chaff for not doing it right.

When you compare that to the beautiful simplicity of Open Source communities, it makes you wonder how anything actually gets done? We’re so afraid (fear) of being wrong or doing the right thing without the right approvals, we let the process hamstring us from fixing the problem. We don’t know (uncertainty) what the right thing is anymore, so we do nothing. And in the end, we’re not sure (doubt) if we’re of any use at all. (I think my premature grey came from this job, and if I leave, the first thing I’m doing is dying my hair neon blue.) Plus, to make matters worse, they told the entire company about the security hole, so everyone knows, and they can see we didn’t close the tickets. It’s a mess.

But really that’s not what FUD is about. Nor is it what this post is about. Neither way is perfect, and both are flawed in different ways.

You see, what Open Source nailed in one was that we should be aware of the dangers, and work together to make it better, not feed the fires and run around terrified about what’s going on. A little fear is a good thing. It clears the arteries and is good for your heart. If you’ve never had a moment where the blood drained out of your face because you made a mistake, you’re not trying hard enough. We all live with uncertainty and doubt, too, and inherently these are not bad things. What is wrong is allowing them to have complete control over your actions to the point of inaction or consistently making the wrong choice that you know is wrong.

Make the right choices. If a bread stick is on fire in the toaster, take it out and make that extra step to sort out who put it in there. Treat everyone as a team member, a fellow hero. You see, if we give in to FUD, we cripple ourselves, much like corporate america does every day with miles of red tape. But if we don’t, if we accept our fear and move forward, we can get past it and make better products for everyone. And that’s a great goal.

If you think you, or a friend, may have been hacked, please go to Sucuri and run the free scan for your website.

19 August, 2011
Backup Where You Belong

I’ll make it quick: At the end of the day, there’s only one person who’s responsible for your backups, and that’s you.

Here’s the deal. WordPress does not have a 100% backup everything tool. Neither does Drupal nor Joomla.(All three of the big guys have plugins that can do this, don’t worry, I’ll get to that in a minute.) In fact, I don’t know of any app on the web that does. Even though Google says “You own your data!”, if you use their tool to download everything, it’s not in a form you can just slap back on the web. Their backups remain tied to themselves. You get the data, but then you have to parse it.

This brings up a bigger question, though. What is the point of a backup? In a worst-case world, your backup is to save your bacon for when you screw something up. It’s to restore from a crisis or to roll back a bad change. So why aren’t these sorts of things built into applications?

When you think about it, they’re not built into any application. From Microsoft Word to your favorite Twitter app, if an upgrade breaks something, there’s no ‘roll back’ option. You can uninstall and reinstall, but most of the time that means you have to reconfigure all your favorite settings.(This is actually why I try to make as few special config changes as possible.) Yes, in Microsoft Office, you can save your ‘document’ in total, but that isn’t a direct analog to Web Publishing, because there’s far more than ‘just’ your book, there’s all those settings and preferences. If you’ve ever tried to copy someone’s preferences and settings from one computer to another (and you’re not on a Mac, who makes that shockingly easy) you know what I mean.

The best backup tools are things like Microsoft’s cloud backup, or Apple’s Time Machine. Both make a massive copy and then incremental updates to your entire computer. They are, as we say, OS (Operating System) backups. All your documents, all your applications, all your settings, are backed up. No individual application has ever bothered with this so why should a web app?

The argument goes that you should be able to pick up your web app and put it down on another server via exporting. I can think of one app off the top of my head that can do that: Cpanel. I’ve never tried it myself, but I’ve been told it works pretty well. Still, Cpanel actually falls under the weird realm of operating systems, as it’s really a server management tool. It’s where you logically expect to see things like your backup tools, db access, etc etc and so on and so forth.

In short, it’s the right place to see your backups made.

How do you backup a webapp?

Step 1) Backup ALL the files on your server.
Step 2) Backup all your databases.

That’s kind of it. For most well written apps (WordPress, Drupal, etc) to ‘restore’ these backups, you just copy them back up. For the database stuff, you may need to make a second database and edit your files to point to that DB instead of the original, but it’s pretty fast. Professionally, we have one-click rollbacks installed on databases, but even then, we tend to go ‘Oh, that didn’t work.’ and rename the NEW DB (databasename_date_BAD) and re-upload the old one. Why? Because it works. When the DBs are too big, we have incremental backups and rollbacks set up. Files ditto (actually for files, we ALWAYS have step one ‘make a copy of the old folder structure…’ and the rollback is just renaming things).

We rarely rely on the applications themselves to perform these tasks for one simple reason: They’re BAD at it.

I’ve always been an advocate of the right tool for the right job. A web app is good at its job. A backup tool is good at its job. The two may cross, but there’s nothing wrong with using a backup tool to make backups and a writing tool to write. I don’t use any plugins on my apps to make backups, I do it via the tools built into my server that are, expressly, for making backups. Ditto my computers at home. I know what the right tool is, and I use it.

11 August, 2011
Understanding Zero-Day

If you run a website or work with computers much at all, you’ve heard the term ‘Zero-Day Exploit’ and you probably had no idea what that meant.

At its heart, a “zero day” attack or exploit is one that happens before any of the developers are aware of it. It’s pretty straight forward, in that the attacks take place in that window of opportunity between code release and code patch. Logically, you’d think that all exploits are ‘zero day’ because a programmer would never release a product with known vulnerabilities. Right?

Wrong.

We already accept the fact that human beings are not perfect and thus, by extension, neither is our code. We cannot make every product work on every setup, otehrwise there wouldn’t be browser and OS wars. Keeping that in mind, we have to accept the fact that there will always be security holes in code. And sometimes we developers are well aware of them, but consider them acceptable risks. This means that when a vulnerability is plastered as a zero day, the question becomes ‘By whose calendar is this a zero day exploit?’

If you found a zero-day flaw in a product, the ethical thing to do is privately communicate with the product developers ‘Hey, if I do this, I can get access to THAT.’ At that point, the product developers should take the time to craft a fix and quietly push it out to everyone. The public often isn’t told about this until the patch is written and available, and even then, details are withheld a few days so that, during the critical time it takes everyone to upgrade, people aren’t exploited further. This also allows people to apply one patch instead of 17, as multiple fixes can be wrapped up into one install.

Of course that’s a perfect world scenario. There are multiple cases of exploits being announced in the wild before a fix has been made. Sometimes it’s a case of an over enthusiastic reporter, but also sometimes the people who report the bug get mad at how long it takes to fix it, and release the information in order to speed up the process. There are unprintable words for those fools, and the fact that they can’t understand how they’re making the situation worse is sad.

By its nature, an exploit no one knows about is the one you can’t protect yourself from. That’s why vulnerability disclosure is such a touchy subject. Sometimes the fixes are really easy, but more often they’re not. Like a vulnerability exploit in your car is the gas tank. Anyone can walk up, unscrew your fill cap, and pour in anything they want. That they don’t has more to do with the fear of retribution than anything else, but they certainly could. Also vulnerable? Your mail. I can’t tell you how many times I see the mailman leave the cart on the sidewalk while she goes in to deliver our mail. Someone could steal the mail, but rarely does that happen.

In 2008, there a DNS cache poisoning vulnerability was discovered.(ZDNet – Has Halvar figured out super-secret DNS vulnerability? by Ryan Naraine | July 21, 2008, 2:12pm PDT) The details of the exploit itself are inconsequential to this story. When the vulnerability was discovered, the folks ‘in charge’ asked for a thirty-day embargo where no one would ask about it or talk about it, to allow the code to be patched and deployed. This radio-silence would end with a news release and explanation. This did not work as well as one might have hoped. (ZDNet – Vulnerability disclosure gone awry: Understanding the DNS debacle by Ryan Naraine | July 22, 2008, 7:09am PDT) People accused the organizers of performing a bit of media hacking (i.e. social hacking) and spinning the news to make a bigger impact for themselves. Essentially, they claimed there were no altruistic reasons to keep the lid on the issue.

When you seen a report of a zero-day exploit, the important thing is not to panic. Firstly, check to see if there’s already a patch. Secondly, remember that just because you’re vulnerable does not mean someone’s spiked your gas tank. Thirdly, accept reality for what it is and know that you’ll be impacted at least once in your life, and that’s okay.

If you know how to recover from this, you’re better off. But that’s another topic.

1 July, 2011

Tag: administration

All Subs Are Equal

TimThumb and the Pseudo (D)DoS Effect

United States Patent: 8010458

TimThumb, Heroism and FUD

Backup Where You Belong

Understanding Zero-Day