Half-Elf on Tech

Thoughts From a Professional Lesbian

Author: Ipstenu (Mika Epstein)

  • MediaWiki – All Powerful, All Annoying

    Don’t get me wrong, I love MediaWiki. It’s ‘overkill’ for what I need, but then again, I wanted a stand-alone ‘encyclopedia’ where primarily text based articles were listed, without the ability to comment. And until someone can trim WordPress to run as fast as MediaWiki, I’m sticking with it. Well, that and they need an ‘import from MediaWiki’ tool, cause at 700-odd pages, I’m not doing it by hand. It’s a static website, and it does it’s job well.

    But right now, and every time I need to update it, I hate it.

    I don’t mind using command line to wget the latest version and unzip it, overlaying the new files atop the old ones. What I mind is having to manually visit the pages for all my extensions, and determine if I need to upgrade or not. It makes me wish for WordPress with the happy ‘Hey, that plugin needs updating!’

    See, there’s no admin ‘side’ to MediaWiki, like there is for WordPress, or ZenGallery, or anything else I run on my sites. MediaWiki is for the hardcore people who don’t mind getting their hands dirty. And as a user, I think this is the real problem with the whole thing. Until they make a user friendly admin side of the whole thing, MediaWiki will remain used by the nerdy, the geeky and the techie, rather than the whole world. Part of why WordPress became so popular is they made it not easy, but easier to run your own blog. It’s still got problems, sure, but they made it so you could easily learn how to manage your own site.

    And then there’s MediaWiki.

    MediaWiki sucks to admin. Like today I found out I could turn on File Caching. That’s great new, I think! I use it for my gallery and my blogs (runs faster among other things). Except that, unlike WordPress (where Donncha’s freakin’ amazing WP Super Cache can clear out files on a scheduled basis) or ZenPhoto (where it runs once a day, or whenever I press ‘clear!’), MediaWiki has no cache expiry. That blew my mind, but seeing as MediaWikis are ‘mostly’ static content, it makes a little sense.

    So I turned it on and ran $php maintenance/rebuildFileCache.php which force caches everything. All at once. This is awesome to get your site ‘started’ and all told, it took up about a moderate, but not huge, bit of space.

    Also, I was told ‘When you edit a page, the cache is refreshed’ except I did, and it didn’t. Then I was told ‘Add this to your page URL and it will prompt you to recache.’ (this being ?action=purge) except that didn’t either. If I was logged in, it did nothing. If I was logged out, it did, but then I went back and it was still the old page. Finally I sorted out that the cache pages had to be owned by ‘nobody:nobody’ (this isn’t too weird, BTW). The problem NOW is that if they were owned by that, then the script rebuildFileCache.php didn’t work!

    So, great, it now works, it now flushs when I edit and save a page. If I run the rebuild command, I’ll have to manually go in and chown the files to nobody, which annoys me, but I have godlike access to the server and I can always fix it. But what if I want to delete everything in the cache? Basically I have to dump the entire folder. Which is annoying, but at least it’s working now.

    Why would I have to flush the whole cache? Because I make a formatting change, let’s say. Also, I have advertising on my sites. How does this get affected?

    In the end, I’m going to keep the cache running for a month, see how it goes. But it still annoys me how much of this is lacking because of no admin ‘dashboard.’

    Then again, that’s MediaWiki. Function over form. All powerful, all annoying.

  • But If, Baby, I’m The Bottom, You’re The TOP

    Earlier this month I talked about how my server was acting wonky and how I fixed it using, among other tools, TOP.

    This week I was chatting with a fellow about CPU usage and his site. He runs a rather large WordPress blog and the database is about 500 megs. As a comparison, this site, with about 500 posts, is under 5 megs, and my big site, with thousands of posts, comments, and a forum, is 10 megs. The biggest site I run on my server is 850 megs (just down from 910 after some clean up). The difference between his site and mine is that his is slow and he knows it. As we discussed ways to speed it up, I had some thoughts on WordPress and how, at a certain point, you’re going to need to dig into the guts of your server and learn TOP.

    The ‘problem’ with most ‘How do I make my WordPress site run faster?’ tutorials, as I’ve seen it, is they address surviving the digg effect. That is, they talk about how to deal with having a high volume of traffic on your site and, for the most part, you can make it with just adding caching plugins.

    Once your site gets ‘big’ or ‘popular’ you’re going to have to move off shared/cloud hosting and over to your own server. For most of us, the first step is a VPS (Virtual Private Server). Shared Hosting means ‘You have an account on a server with a hundred other people.’ It’s great for small sites, inexpensive and easy to use. The problem is you could have terrible neighbors, who use up all the CPU. Think of it like those old New York apartments where someone’s a jerk at 5am and uses up the hot water so you, at 7am, have none. Yeah, it’s kind of like that. That’s the day you think ‘I want a house!’

    Only, well, we’re not all up for houses just yet. A house would be a dedicated server, where it’s just you. Cloud hosting, which I touched on earlier, would be the college dorms of webhosting. It has a lot of benefits for the really small sites, and actually some for large sites, but I’m not sold of their overall usefullness yet, so I’ll talk about them some other time. What I want to talk about are Virtual Private Servers, the condo-sub-leasing (or rent-to-own maybe) of website hosting, and how the new VPS user should really get on TOP of things (sorry, bad pun) to make their lives easier.

    TOP. Well ‘top’ really. Unix commands are generally all lower case like that.

    The top command is a system monitor tool that outputs a list of processes. Have you ever seen Task Manager in Windows? It’s kind of like that tab for ‘Processes’ that you look at and run away from. The default view of top is by percentage of CPU usage and the “top” CPU users are listed. See? The name made sense. You can also see how much processing power is being used, memory hogs and other cool things. Most modern Unix-systems let you sort the list, colorize it, etc, though you have to be command line savvy.

    Here’s what top looked like for me about an hour ago.

    top - 12:44:44 up 126 days, 23:13,  1 user,  load average: 0.12, 0.17, 0.17
Tasks:  91 total,   1 running,  90 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    524288k total,   358248k used,   166040k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15616 nobody    15   0 94540  65m  20m S  1.0 12.8   0:00.27 httpd
12261 ipstenu   16   0  1908 1012  780 R  0.4  0.2   0:00.30 top
[...]
28630 root      16   0  107m  86m 1096 S  0.0 16.8   1:06.30 /usr/sbin/clamd

    I wanted to point out clamd, which has been the bane of my existance. Thing won’t DIE. I ended up going in to /etc/exim.conf and manually commended out the clamd line (and restarted the service) to finally get it gone.

    But top, as you can see, has a freakishly large amount of information. My server is doing fine, at this point, so I don’t have a whole lot to show you. What you can see right away is that I can tell, with a glance, what’s going on. I could see, though and at this point I have a ‘nobody’ process. That just means someone’s accessing my website. No, really! That’s good! The CPU and memory usage seem high, but they vanish in a second. Basically, someone rang my doorbell and for that brief moment, electricity was used. The next thing I see is the top command, which is run by me (hi!) and down the line is that idiot, clamd.

    I actually scan top a lot at work these days, trying to understand what’s causing issues. It’s good for ‘right now!’ things, but not so much if I want to see what started a strange spike a couple hours (weeks) ago. For that you need a whole mess of tools.

  • Sucking Clams, Kosher Style

    ClamAV is an tool that you put on your server and it detects malicious software. In short, it’s a server virus scanner and most servers use it to scan email for viruses. Now those of you who use stuff like McAffee and Norton and other virus scanners for your email, you may not know that servers also scan for that stuff as well, and try to kill the emails before they ever get to you! Yeah, think about how many emails with viruses you get. Personally, I’ve never had a problem with viruses and not because I use a mac. It’s because I pay attention to the content and context of an email before I open any attachments.

    But this is about ClamAV and server-side scanners.

    The story starts with my twice a week check of my server. I like to keep tabs on what it’s doing, how it’s doing, what’s going on, etc etc. I was a little surprised to see my server load spiked. Server load is sort of how you know how hard your server is working. A high load means its looking at a lot of work. A low load is ‘better’ but you have to admit that you’re going to have SOME load, so you may as well figure out what’s a good load for you. I’ve had problems with WordPress and right now I’m using WP Super Cache (See “I take it back. WP-Super-Cache is a Super Hero” from September 2009).

    The point is, I know that a spike like this is okay:

    That spike there was when I ran a small upgrade. You’ll notice how after the moment, it drops back down and has a happy nice day? That’s how things are supposed to work. A spike with traffic and then everything’s happy again. Great.

    So what does this mean?

    Yeah, I took a look at that, paled, and asked myself ‘What in the four hells is going on!?’ I did the logical thing and looked at the date and time. Noon on Monday I’d made a change to the firewall, moving from the perfectly acceptable, though harder to manage (no GUI), APF Firewall to CSF. That move was a TEENY bit on the spur of the moment, as I wasn’t having any problems with APF per se, but I was being hit up by a lot of spammers and my usual attacks of http:BL and Bad Behavior weren’t cutting it. They’re front end fixes to the ongoing spam problem, alas. I hate spammers.

    Worried that my new firewall was ‘bad’, I started to Google if CSF caused high server loads. And found nothing. So I went back to the beginning and checked top. Top is a unix command that you use to see what’s using up resources on your server. It’s like Task Manager for Windows, but it’s a lot more informative. Top lets you see details and sort and basically when you want to find out what ran off with the spoon and killed your server, baby, I’m the bottom and log on to top. Top showed me, interestingly enough, that ClamD was using between 70 and 90% of my resources. On a slow week, like the net generally has for entertainment sites between Christmas and New Years, that’s not really a problem. There’s not a lot going on with the sites I host right now, the extra CPU usage wasn’t a problem. Come back on January 20th, though, now that’s a problem.

    But the thing of it is, back in September, I optimized my server and I remember reading on multiple places that ClamAV and ClamD use up a lot of resources and people turn them off. So I did.

    Isn’t that much nicer?

    The real question, at the end of the day, is if having ClamAV turned off causes more problems than having it on? So far, no one’s breached my servers, though that’s a function of my firewalls, and SpamAssassin seems to be taking care of the spam emails, which is where most viruses come from in my experience, unless the server’s hacked, at which point I’m kind of screwed anyway. But what I find myself wondering now is if it’s dangerous to not be using ClamAV or what. And I don’t have an answer to that yet.

  • I take it back. WP-Super-Cache is a Super Hero

    cupspikes I’m going to be upfront and admit that I’ve never actually liked this plugin. A very large part of me wants to side with Matt Mullenweg in that if you have a good server, configured properly, with a decent host, you should be just fine. Also, it doesn’t really work well with my favorite anti-spam plugin, Bad Behavior, which stops 99.999% of my spam cold. But. Over the years of running a vaguely popular fan site, I’ve been nailed by service spikes that killed me and everyone else on my shared hosting setup (multiple websites, not all connected, sharing a virtual server). At one point, I had to offload ‘news’ to LiveJournal, but since then, I’ve pulled it all back to WordPress, moved to a virtual private server (VPS, just me and my sites on a virtual server) due to the need for better support, and I was kind of complacent. Things were trucking along just fine, we had some major news that were handled without a blip, and I thought I was cool.

    Yesterday I had to cycle the HTTP service three times to clear things up. The first time, someone was using a really old URL (for a part of the site gone 2 years now) and, when it didn’t give them what they wanted, they kept hitting it. I blocked the IP address and we were fine. But. Then the news that I had some new, cool, information hit, and suddenly I was spiking like mad. I checked my stats, trying to see what was the culprit. The gallery is pretty tolerant of these things (though I have turned on the Static HTML cache right now) and while I did have some hefty images (1 to 2 MB, I usually try to keep ’em to .75 megs), it wasn’t ZenPhoto borking.

    No, my poor, poor WordPress was having a heart attack because I’d gotten myself crosslinked from a couple high traffic sites. How bad? Well that spike on the graph below may explain it.

    spiked

    The first thing I did was tune the server. Actually, I’d done that months ago, dropping my memory usage from 77% to about 60%, but now I went in to see how well that was working. There was a little more I could do, so I optimized a couple more settings and things eased up a little. Not enough. I scrubbed the CPU usage, too, and normally we never spiked over 1 for a load average, but that wasn’t working yesterday. Sidebar. CPU Load is a very bizarre thing to most newbie server admins, and I’m not great at sorting it out myself. Of course, I know that a ‘good’ load is anything around .3 and a ‘bad’ load is something like, oh, 9. And yes, I hit 9 yesterday on my 8 processor VPS box. I’m not going to explain it here, as I’m still learning and I’m sure I’d get it wrong, but the gist is that you don’t really worry if your load hops up to 1 or 2 for a short amount of time. When it stays there and is spiking at 4 or 5, however, you need to pay attention.

    What kept happening to me was that it would spike up to a load between 5 and 9, and the HTTP service (the bit that serves up webpages) would scream and fall over. Email, FTP, shell access and the rest were all okay, though, so I knew the server itself was fine. Thus I deduced something was sucking up the load and I knew I had three choices: JFO’s blog (it’d happened before), JFO’s gallery, or YTDaW. While I host YTDaW, I don’t actively admin it in any authoritative stance. The only ‘mod’ work I’ve done is turn off email alerts for people who are using non-existent emails (and then, only when I’m tired of getting their bouncing email). Devon pretty much keeps me on tap for server admin and security stuff, and I do my utmost best to keep my hands OUT of the pie. It’s her baby, I’m just the tech.

    And while they’re using a pretty old version of the forum software, it’s secure enough and solid enough that I didn’t think they were the culprit. The evidence (heh) supported that theory, so I went to look at JFO. It was definitely my old girl, and right away I could see that we were getting a lot of traffic from new users. Four times the traffic. Before you could say ZOMG! I was on Google Analytic and Woopra, checking out who the hell was hitting my site and the answer was surprising.

    Everyone. (Well, mostly FaceBook, AfterEllen and Twitter, but really, it was all over.)

    I’d accidentally broken news about three hot topics within a couple hours, and now everyone and their mother wanted to see JFO and, as many people have mentioned, WordPress was hemorrhaging under the ‘digg’ effect. Basically it was trying to serve up dynamic (generated on the fly) pages to too many people at once. If I was using static HTML, it would go faster, but WordPress doesn’t do that. Except … except it does if you use WP Super Cache.

    no10 As I mentioned before, I don’t (didn’t!) like that plugin. I want my app to behave correctly without it. I mean, the PM of Britain uses WordPress! I was sure they don’t need caching. They probably have a rack of servers on a co-located cluster. Except I viewed source and they were using it. The Library of Congress wasn’t, though, and neither were The Speaker of the House (Nancy Pelosi) or the Army. Honestly, I wasn’t sure how to take that, but after four hours of babysitting my server, I took a plunge and installed WP Super Cache for the fourth time.

    The first few times sucked, I admit. It was a lot of massaging and manual crap that, while I’m perfectly capable of doing, I didn’t like. This was easier. A chmod, an install, a click, another chmod and then I was done. And guess what? My loads dropped from an average of 3.45 to one of .35 by morning. On top of that, my memory had one spike since I turned it on, and that was right when I was running backups and the like.

    memoryspike

    So I’m keeping it on for now, especially with what I expect tonight, but I think that I can say … yeah. WP-Super-Cache does what it says.

  • Every Site should have a Favicon

    wikipedia-favicon Imagine summing up everything a website is about in a 16x16px square. That’s the goal of a favicon (short for favorites icon). Pretty much every site out there has one, and it’s a devil of a task to make one that looks appropriate, identifiable and understandable in such a small space. As much time as I spend tweaking a design I spend on a favicon because they are that important for the look and feel of a site. A site without one is nearly naked.

    Back in the days of IE 4 (yeah, 4, so 1997), Microsoft hit upon a great idea. If you made a teeny picture and saved the file as favicon.ico in your html root, their browser would pick it up and be the icon on your bookmarks menu. It didn’t take long for people to figure out microsoft.com was doing this, and they began implementing it all over for every site they could. As people got smarter, they figured out how to fake it, so you could have a different favicon for every page, just by manipulating the head of your html document.

    Back in the day, you had to use .ico (Microsoft Icon) files as your favicon, but these days most modern browsers pick up .png, .gif and .jpg happily enough. This allows people to make animated favicons, which need to be shot and killed. For maximum compatibility, though, most people still use .ico, since IE doesn’t like the others. Or it didn’t. Someone on IE 8 will have to check.

    The real problem boils down to size, for most people. At 16x16px, you don’t have a lot of room. This site actually has a non-recognizable icon (it’s the Xena/Gabby picture). Technically you can go up to 32×32 for an image, and I have one that’s 240×240, but in the end, they all render at 16×16 on 99.999% of browsers, so looking good at that size is your goal.

    If you think I’m being silly, about a year ago, Google changed their favicon and admitted that it wasn’t final. Right away they basically started open submissions for a better one. When they changed it in January, it became the favicon heard ’round the world. Eventually, Google stepped up to explain the change. It’s important to have an icon that matches your site, as Google explains, as well as a unified look for all aspects of your design. Should you have a different look for each app on your site, or an all in one? How does it affect the other aspects of your site, like the iPhone’s new icons for saved webpages?

    These aren’t simple answers, but to explain how I go about it, here are some favicons that I have made and use out there in the world. Not this site’s though. I need to come up with something better for it.

    jfo When I moved JFO from orange to green a year or so ago, I made a new favicon to reflect the design. The image is a cropped shot from the original header (which is now a full color photo, but still), and is a close up of Jorja’s face. It’s JUST recognizable as Jorja, I think.

    jfo2 Alternately, I came up with this image, which is a copy of the shot used on the header currently, done in greens to match the site. In a way, it’s both more and less recognizable, as the image is harder to make out (it’s a head and shoulders) but as it’s the same used in the header, people might make the connection. I’ve yet to use this on a live site, but it shows up on my test sites right now.

    For the website ‘SCA Jews’, I had gone with a slightly eastern feel of a website, that evoked both the idea of camping with the concept of days gone by. Evening Sun came from spectacu.la, and took minimal editing to fit my plan. The problem was I had no favicon. Originally I put a little sun up there, but then it struck me that the ‘meaning’ of the site was to promote the meal plan “Meals on Camels”. What better way to express this than with … a camel.

    yeast I also helped design (or rather optimize the design for) my friend’s site, The Yeast I Could Do. She had no favicon and I spent a couple hours scrounging for something bread-ish, and eventually picked this one, even though it’s questionable. It does look a bit like a loaf of bread, and she recognized it, so I think it went okay. In it’s .ico format, it has a transparent background.

    ponywars Finally there’s this one. Pony Wars is a joke site I made up with a friend for a “My Little/Pretty Pony RPG”. I mocked up the site because I was bored one day and finding an icon for it has been a bear. In the end, I went with this 33×33 (yes I know) icon of a pink pony. It doesn’t scale very well and looks weird on the site itself, but it’s a hard icon to shrink.

    If, in the end, you’re stumped at making one, there are a lot of favicon collections out there to help you. Be warned, they can take a LONG time to load:

    What are your favorite favicons?

  • I Haven’t Got Time For The Pain!

    Carly Simon and you should get the joke here Two months ago (give or take) I mused over photo gallery options for my sites. For Ipstenu, I’m now using WordPress and treating it like a photoblog. For JFO, however, I couldn’t answer it that easily.

    I really do like the Gallery project. I do! I learned a great deal about photography from it, and I’m thankful for it. But. I needed to move on as a user, a developer and a photographer. On that last one, I’m not a profession one, I’m just a goofy girl with a camera who likes to remember where she’s been. As a user, Gallery2 did the job well and without major issues. As a developer, it made me want to cry. Many times. Once I had to log into my friend’s server to fix his install. That just whomps.

    Even the developers admit that Gallery2 suffered from bloat:

    The code base is too complex and over-engineered because it was designed to fix every single thing that was wrong with Gallery 1 (Second System Effect) leaving its scope hazy and broad.

    The whole idea of it was “Your photos, your website.” And personally I love that. I hate having flikr or picasa in charge of MY photos. Let alone FaceBook. I have a blog on my domain for that same reason. But Gallery2 was too much. I never used half of it and it was 16+megs at its slimmest install. That the developers agreed with my feelings delighted me. And the Feature List was also exciting. As soon as G3 popped out, I grabbed a copy and started playing.

    With each version of Gallery3’s beta releases, I would get excited and then disappointed. Excited for the new toys and disappointed for how the overall effect felt. It just felt wrong for me. It wasn’t really Web2.0, even though it was, and the usage felt off. It didn’t make intuitively as much sense as G2, though it was still far better than Coppermine (which frankly I hate, and I know more people who argue with it than anything). At first I thought it was because I was so used to G1 and G2, but then I realized that over the last 10 years, I’ve used so many different systems that I’m fine with subtle differences. I’m savvy, I’m smart, I can code, so why did G3 feel wrong to me?

    It was too hard. Too much was built in and not plugable. Too much was hard coded in itself. Theming was impossible in the first release, and way too hard in the third. Understanding the theme system in G2 was easy, though implementing it was hard. Understanding it in G3 was hard and implementing was horrific. And before someone reminds me, AGAIN, that this isn’t even a beta product but an alpha, quite frankly that’s not an excuse. The basic things you need to be able to do with a first public release (be it beta, alpha or whatever) is to use it: Upload photos, change options, theme. That’s it. Those are the three things at it’s most basic that photo gallery software has to have, or you may as well be using an off-site solution.

    And while I may sound like I’m ranting, I’m not. I’m sad and frustrated and … You know, I really like Gallery! I really do. But it was starting to feel like Movable Type. They made a big shift and suddenly I wanted to know who peed in my coffee. The code felt wrong, it felt klunky, it felt raw. It was like starting over, and I didn’t like where it was going. And I realized the fact was that I was going to say goodbye to an old friend.

    Personally I’m all about the simplest, best, tool for the job. I wanted a way to update news on JFO and, when that was ALL I needed, I used CuteNews. When I realized the site was going to need something more, I weighed my options, tested software, and decided that while WordPress was a bit of overkill, I knew how to support it and customize it to be what I needed. In the end, that proved to be a perfect choice. When I had a forum (the first time around), it was IPB, which I liked, but it always felt too big. Now I use the very basic bbPress and it’s what I need and nothing more.

    If WordPress had PhotoPress, I’d probably have snagged that. Instead, I shopped around. I installed Coppermine, again, to test. I put up G3-alpha3 and then 4. I went to WikiPedia and dug out the compares and ended up in a head to head battle between ZenPhoto and Gallery3.

    ZenPhoto won by feeling better.

    Seriously, it’s asthetics at this point. There are only two features I miss: Being able to re-upload a picture and keep it’s MetaData, and having ‘new’ images show up with a different background color. But I can live without those.