Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: software

  • Torrenting

    Torrenting

    I torrent.

    Look. There are hundreds of legit reasons to do this. I’m not going to argue if it’s right or wrong, because we all know that taking something that people charge for, without paying, is called stealing. Let’s put that aside. I’ve got a torrent I want to download, because downloading this way will be faster than a normal FTP of a 1G file.

    The first thing you need is a Torrent client. I like uTorrent personally, as it’s small and easy for me to use.

    The next thing you need is a torrent file. Let’s say I want to get the DRM free music from SXSW. I could download them all one at a time, or I could use a torrent from http://www.sxswtorrent.com/

    2005-sxsw

    That’s 2.6 gigs of music. And you’d think this would take a long time to download. You’d be wrong. What you download is “SXSW_2005_Showcasing_Artists-Release_1.torrent” and open that tiny file in your client.

    uTorrent lets me select what files I want to download (all of them today):

    uTorrent

    I click okay, and it starts to download. This is where it gets fun. The more people who are sharing this file, the faster my download goes.

    utorrent-dowloading

    Here it is, trucking along at 1M a second and it’ll download in about 45 minutes. Just enough time for me to take a shower.

    The magic in the torrent sauce is that instead of FTP where it’s one person asking one server for one file, this is a million people asking a million people for a file, as bits, not a file. So I can get one snippet of a file from Bob in Minnesota, and another from Arman in Atlanta. The more of us who download, the more of us who are sharing what we just downloaded!

    But torrenting is really my favorite way to share large files to the masses. And yes, when I’m done downloading, I leave it up to ‘seed.’

    I am aware, by the way, that as of 2010, SXSW was a little iffy about distribution via torrent, but it’s been three years and no one’s shut the site down. They’re all well aware of it so if no one’s been slapped with a C&D by now, they’re all okay. Anyway, all these songs are available to download off SXSW’s official site. The issue was with the distribution, which I can understand. I try really hard to only download legal files from legit sources, because I appreciate the value of the work that went into what I’m enjoying.

    Now if you’ll excuse me, I have some new music to listen to!

  • Cloud Experiment

    Cloud Experiment

    While I’ve mentioned that I don’t ‘get’ CloudFlare, I took the time to buttonhole the guys at DreamCon and explain my issues. Many thanks to Maria Karaivanova for her presentation at DreamCon, too, which helped me a lot.

    Now, in so far as a ‘traditional’ CDN (where they host my images) goes, I don’t need it, but as a Cloud Proxy, I both understand and like it! The deal with a proxy is pretty simple: It’s an extra firewall between your server and users. Why is this faster? Because they have more servers than I do, which means they can handle a DDoS better than pretty much anything on Shared Servers will ever be able to do.

    Keep in mind, a VPS can handle a lot of this on it’s own. I could install Varnish and use that for caching, but it wouldn’t give me the ability to have multiple servers serving my content, and that’s what I’m looking for with my experiment here.

    It would be remiss if I didn’t note the more well known alternatives: Incapsula (starts at $19.99/month), Sucuri’s Cloud Proxy (starts at $9.99/month), MaxCDN (starts at $9.99/month), and Fastly (starts at $50/month). CloudFlare starts at ‘Free’ but its first paid offering is $20 a month for one website, $5 for each additional.

    On My Server

    I use ConfigFirewall (CSF) so I had to add in the CloudFlare IPs into csf.allow and csf.ignore. Yes, both, otherwise I got weird alert. This is pretty easy, though.

    Next I installed mod_cloudflare because I wanted to preserve the IP address without having to muck with plugins on everything. This particular site is my ‘Not all WordPress’ site after all. The catch is if I do it all manually, I have to redo it every time I upgrade via EasyApache(Don’t judge me). I already have to do that for PageSpeed. That said, cPanel suggested I read Installing mod_cloudflare on cPanel, so I did that and then ran EasyApache:

    Screen Shot of EasyApache

    So that was easy! By the way, TL Tech is one of my standard resources. They have a lot of tricks, and I’ve bookmarked ’em.

    Finally I checked out if there were issues with PageSpeed and CloudFlare. CloudFlare says no, but indicates the redundancy. That’s okay. I did an extra step of telling PageSpeed to not modify caching headers, as that’s something we did for DreamHost and Varnish (DreamPress!). Just add this to your .htaccess section for PageSpeed.

    ModPagespeedModifyCachingHeaders off
    

    On CloudFlare

    whohostsThis was straightforward. Follow their directions and it’s fine. I went for free, and fiddled with my Security Settings a lot. I hate captcha. And I know, I knooooow, the users for this site will cry if they get hit by one, so I turned my security to “Essentially Off” – This is the only way to get rid of Captcha. Sad panda. I also turned “Browser integrity check” on for now.

    In Performance Settings, I made Caching level “Simplified” and left the rest as default. Then I set up PageRules for WordPress and my other apps. You only get three rules with free (and 20 with the first level plan) so I made sure to free up the admin tools.

    On my webapps

    Last up, tackling purging caching. I’m far more familiar with this now, as I support a Varnish plugin that does much the same (and I did consider installing Varnish). The official CloudFlare plugin, for some reason, only serves the same purpose as mod_cloudflare, in that it restores IP addresses. But what I really want is a way to purge my cache with a new post. Pretend I’m saying this in that voice I used at WCSF… there’s a plugin for that: CloudFlare Cache Purge.

    Sadly there isn’t a similar plugin/extension for my other apps. And this is why I ended up at my current conclusion…

    Current Conclusion

    Sadly, even after letting it bake for a few days I determined it wasn’t quite right for me. Everything worked, and if my site was more static, it would be perfect. But this brought up the same problem I’ve had with all caches: my dynamic content gets hurt.

    cachingWhat is static that I can and should cache? JS, CSS, font files, images. What is not static? Blog posts, comments that are happening all the time, fast and furious. A gallery that needs to update. A wiki that has a deadline. Worst of all, it prevented two of my apps from being able to make their own ‘static’ cache in the background. Now really that means I shouldn’t have to make my static cache at all, but this brought up another issue. Coordinated pushes of content, where four separate apps update 1-3 pages each at the same time means I need to be able to purge those pages, right away. And right now, there aren’t extensions to do that.

    Of note: I noticed the exact same problem with Fastly and Varnish, so it’s not just CloudFlare, it’s a function of how these things are supposed to work.

    What would I need to make these desirable? Basically I need a way to purge my cache on the proxy efficiently, quickly, and selectively. Now that I work on the Varnish Cache at DreamHost, I’ve seen how deep the rabbit hole can go with this, however, and I know fully how hard this is. Proxy Caching is not for everyone. When you have dynamic content that changes for logged in users on the fly, it’s a pain. I mean, I use PageSpeed to compress and cache CSS and JS, and I have to flush it when I update my site design. Caching your caching is always going to be tricky, unless there’s a simple, one click, way to say “I’ve updated these pages, please purge them.”

    We’re not there yet.

    Recommendation

    CloudFlare is pretty awesome, actually. If you’re ‘just’ running a blog on shared hosting, I would seriously consider using it, especially in light of the various DDoS attacks out there. A cloud proxy will help you, if you don’t have server level access to tweak mod_security. The fact that CloudFlare gives you a ‘free’ option to test with, without having to give anyone your credit card info, makes it great for experimentation and puts it above the other proxies right now.

    But with all things, keep in mind your personal usage. It’s not just “Does this make my site run faster?” but it’s a lot of “Does this make my usage of my site better?” For me, they win on the first and fail on the second. Maybe one day I’ll change my workflow so cloud proxy, or Varnish, can be the answer, but that’s not today.

  • Tiny Tiny RSS

    Tiny Tiny RSS

    Tiny Hippo Had a Tiny Train
    Credit: Poorly Drawn Lines
    The majority of my ‘I am going to learn this if it kills me!’ lessons come from when I’m just dead frustrated with a product. Today it’s Google Reader.

    I like RSS feeds. They work well for me, I like having them sitting there, waiting for me to pay attention. It keeps news out of my email (which is for communication) and makes sure even if I miss a tweet, I see the article later. The world comes to me. This is also a part of my evil ploy to keep myself at Inbox Zero (current status – Inbox has 7 emails at home, 11 at work). I only leave emails in my queue when I know I need to hang on to them for a good reason, and even then I’m likely to save them off line (I use IMAP) to keep anything long term.

    For the last few years I’ve been using Google Reader because I check my RSS feeds at work (Windows XP, still) and home (Mac), and it lets me sync between the two. Google Reader remembers what I’ve read, and I don’t have to re-read things. But recently Google screwed me over, hard, with their inability to update feeds in anything resembling realtime. Specifically, all my WordPress.org feeds were days, if not weeks behind, and I couldn’t force them to update! What was going on?

    At first I thought it had to do with WP’s recent(ish) server upgrade to nginx, as certainly the problem started around that time, so I asked Otto and Nacin if there was a thing going on. Otto replied that there was, but it was Google. See, Google uses PubSubHubbub, which allows for updates in near-real-time. Sounds cool. If it worked. Now before you say it’s clearly me having the problem, it’s not. I asked around, and everyone who monitors WordPress.org related feeds with Google Reader has agreed: the feeds ain’t in real time anymore.

    I switched to another feed reader and, lo and behold, it worked just fine. Well that sucks. Now how can I handle all this? I could use a dedicated feed reader, but then I’m stuck only reading on one computer, which I certainly could do, but it’s 2012, and I like portability. What sort of options am I left with? After two weeks of experimenting and testing with various web-based readers, I decided that Google really was the best of them, and I was depressed. But I wasn’t defeated. I knew, I just knew, that some clever soul felt my pain and wanted to help me out. And I was right.

    Enter Tiny Tiny RSS. Tiny Tiny RSS is an open source web-based news feed (RSS/Atom) reader and aggregator, designed to allow you to read news from any location, while feeling as close to a real desktop application as possible.

    Tiny Tiny RSS

    Update daemon is not running.On the grand scheme of things, it’s easier to set up than RSS2Email (which I use on another account on this server), but due to me being on CentOS, which doesn’t really daemonize well for users, I had to cron job my feed updates at first. I set it at 15 minutes, after I ran it manually a few times to catch up. There are a few ‘quirks’ that aren’t as nice as Google reader. Like I have to manually refresh the back to get the read/unread counts to work right, and even with define('ENABLE_UPDATE_DAEMON', false); set, it keeps telling me that the update daemon is off. Turns out I also had to delete the /lock/update_daemon.lock file.

    Meanwhile, I dug further into this and found the pretty easy setup for ‘screen’:

    $ cd /public_html/tt-rss
    $ screen -S updaterss
    $ php ./update.php -daemon
    

    And detach from the screen with CTRL+A+D shortcut. Now someone mentioned that this will ‘break’ if the server is rebooted, so I left my cron job running just in case. I’m happy with cron, if it comes down to brass knuckles.

    I’m happy with this, and it’s only been a couple hours. The actually install process was easy, but this isn’t something I’d suggest if you’re the sort who wants a lot of help and hand holding for an app. I’m monitoring my CPU/memory consumption right now, but it seems pretty minimal, so I’m really pleased I have an alternative I like. My wish list is insanely small:

    1. A ‘click to refresh all feeds’ button, instead of relying on cron/command line(I could probably code this myself, just haven’t yet)
    2. Auto-refresh of the page resets the read/unread counts correctly

    And the ‘fix’ for now for those is cron/cli and refresh the page. So I’ll live, quite happily.

    Tiny Tiny RSS lives at http://tt-rss.org but they’re also on GitHub.

  • Alfred.app

    Alfred.app

    My friend Trepmal loves Alfred.app. Since she touts it, I decided I should try it.

    What I Had To Do

    Generally I use Spotlight to search for things. I type in ‘Transmit’ and hit enter and there it is. To make myself use the alt-spacebar of Alfred, I knew I had to hide that from my menu bar. Thankfully, that’s pretty easy. Run these two commands in Terminal.app and the magnifying glass is gone:

    $ sudo chmod 600 /System/Library/CoreServices/Search.bundle/Contents/MacOS/Search
    $ killall SystemUIServer

    Now I start to move the mouse up there, see it go, and know to alt-spacebar it.

    What I Like

    It’s fast and it’s versatile!

    What I Don’t Like

    One of my (few) issues with it is Growl. I just don’t like Growl. There’s something about how it pops up that bangs away at my ADHD/OCD nature and makes it harder to pay attention to anything. Instead, I like apps that change color in my doc, or on my menu bar (think how the Twitter icon goes blue when you have something to pat attention to). That catches my eye in a non-demanding way, which is what I need most of the time. When I need pop-up replies, or audio/visual alerts, each app (Skype or my IRC app) has settings I can refine to give me just what I need.

    Sadly, some of Alfred.app’s code needs Growl to pop up and alert me of things. I wanted a whois extension, so I could type in ‘whois xxx.xx.xxx.xxx’ and get back ‘That’s foo in blah.’ All the extensions I found use Growl. Trepmal asked me what I’d rather use, and I said “I really don’t know.” And I don’t. A terminal window would probably be fine, but the problem is Growl really does fit the bill here. It pops up, you click and it goes away, and it’s not another app running.

    Still, everything else about it I really like. Once I solved my (silly) problem with not showing contacts, which was entirely my fault, I’ve been having a blast.

    http://twitter.com/Ipstenu/status/179683776497061888

    Right now, I’m using it instead of spotlight. I’ve started using the basic calculator, and I picked up the Power Pack for more goodies.

    Suggestions for how best to use it? Lay it on me!

  • Risk Theater and Open Source Testing

    Risk Theater and Open Source Testing

    Audits Are Fun!We make multiple test environments and platforms, testing with hundreds of users.  We perform stress tests (how much traffic can my site take?), and have an obscene amount of checks and balances to ensure that only code that is good makes it into the file product.  We have teams who question every change asking “Do you need this?” or “What’s this function for?”  We audit every update process and ensure that our work is as good as we can make it.  This is all done, we say, to reduce our risk.  Our software, we insist, will be better if we do all these things.

    But the failure rate has not dropped.

    Initially, when a product is released, there’s a spike of failures.  We didn’t anticipate that, or this didn’t work like we expected it to.  Those are not classified as ‘failures’ but as kinks to be ironed out.  Six or seven months down the line, we’ve released another set of itterations to fix the worst offenders and our failure rate drops to a comfortable rate where, most of the time, everything’s fine.

    What if I told you that one in five IT projects was a success?(Source: Statistics over IT Failure Rate)

    What if I told you that all your myriad checks, balances, testing, forms and CYA dances didn’t make anything less risky?

    What if I told you it was all Risk Theater.

    Of course you can do things in a less risky way.  If given the choice between dismantling a bomb in a nice quiet room, where you have all the time in the world and a blast shield, or doing it on the back of a van while being shot at and you only have 30 seconds, everyone would point at that room and say ‘Less risky!’  And they’d be right.  The problem with risk is that there are often, if not always, external forces that perpetuate risk.

    We have to ask ourselves “What is risk?”  We can look at it mathematically.  Risk = {si, λi, xi} – and most of us have no idea what that means.  Risk is not a magical number that says “Defusing a bomb is this risky.”  Determining risk is how we discern how likely something is to happen, and from that, what is the likelihood of an unwelcome outcome.

    Too often risk is defined as risk = likelihood * consequence and safety = 1-risk

    This can misinform: acceptable risk is a consideration of likelihood ANDconsequence, not a simple multiplication with safety as the additive inverse of risk. Acceptable risk and safety are normative notions, changing with situations and expectations, and must be assessed accordingly. (Source: Woody’s Perspective – by Steven A. “Woody” Epstein)

    Risk analysis, for all it’s a mathematical discipline, is just that.  A discipline.  That means the numbers matter far less than you think they do, and if all you do is look at the numbers and say “But we’ve predicted a five point uptime!” then you’re ignorant of the truth.(A five point uptime refers to the claim people make of providing 99.99999% uptime.  The five 9s after the decimal point are feel-good numbers/)  The trick to it all is that variation is something computers are phenomenally bad at handling.  Look at your computer.  It’s what can be best described as a ‘brittle’ system.  If you throw something a computer’s never seen before, it tends to react poorly, because unlike the human brain, it can’t adapt or improvise.  It can’t know “Oh, you meant ‘yes’ when you typed ‘yea’” unless some programmer has put in a catch for that.  On some systems, it may not even know the difference between an uppercase Y and a lowercase y.

    Variation in LeavesVariation is nature.  It’s reality.  It’ll never go away, either.  The point of risk analysis is not to come up with that number to say ‘By doing foo, we are x% less risky.’  The point is to look at the system and understand it better.  The point is to learn.  The act of explaining and defining the process, whatever it is from changing a tire to pushing software to a few hundred servers, is what makes a process less risky.  You understand what it is you’re doing, and you can explain it to someone so they too can understand it, and now you know what you’re doing.  The numbers will come, but they’ll also change over time due to variation.

    We mitigate our risk by understanding, testing and documenting.  But just as you can never have 100% uptime on a system (you have to upgrade it at some point), you cannot excise risk entirely.  On the other hand, we cannot ignore the need for testing.

    A woman named Lisa Norris died due to a software error, caused by a lack of testing.  All the safety checks, the manual monitoring and brainpower failed because the automated system wasn’t tested.  Prior to the automated system going online, the old way was for people to manually transcribe medical dosage.  This was felt to be ‘high risk’ because there was a risk of transcription error.  However nowhere in the incident report were any ‘manual errors’ noted, prior to the automated system being used. We can assume, then, that any manual errors (i.e. transcription errors, the risk the system was meant to abrogate) were caught in-flight and corrected.  The automated system does not appear to have ever been tested with ‘real world’ scenarios (there’s no documentation to that affect that anyone investigating the situation had found).  If they had run simulations, testing with data from the previous, manual system, they may have found the errors that lead to a woman’s death. (Source: Lisa Norris’ Death by Software Changes – by Steven A. “Woody” Epstein)

    There remains a possibility, however, that even with all the testing in the world, that the error that led to Miss Norris’ death would have been missed.  So how do we make testing better?  As long as we’re only testing for the sake of testing (i.e. it’s expected, so we do it), or we follow the standard test plan, we miss the point of dry testing.  Even people who stick by their ridgid test scripts are missing the point.

    Open Source software, however, gets the point.

    Monkeys sans keyboardsYou see, we know we can’t test everything, and we know that we’re going to miss that one variation on a server where code that works a hundred times on a ninety-nine servers will fail on that one where it has a tiny difference.  And yet, if a million monkeys banging on a million keyboards could write Hamlet, then why can’t they fix software?  They can help cure AIDS, we know.  Crowd sourcing knowledge means that you let the monkeys bang on your data and test it in ways you never imagined it being used.  No longer driven by a salary (and that really does lock your brain in weird ways), the monkeys (and I’m one of them), cheerfully set up rigs where we can roll back quickly if things break, and start just using the iterations of software, coming up with weird errors in peculiar situations.

    We always talk about how we want to lower the bar and make products more accessible to more people.  Make it easier for them to use.  In order to sustain that model, we need to embrace the inherent risk of software and teach the users how to correctly perform basic troubleshooting and report real errors.  To often we write our code in a vacuum, test it in a limited fashion, and release into the wild knowing there will be a second release to fix things.  As development matures, we push out more changes more often, small changes, so people are eased into that new thing.  We step out of isolation and embrace the varations of how our product will be used.

    Now we need to get our users to step out of their isolation and join the monkeys.  We can’t make things better for everyone unless everyone is a part of the improvement process.  We must ease these users into understanding that every software product is ‘in progress’, just like we taught them to accept that all webpages are perpetually ‘under construction.’  Our dry tests will never be complete until we can determine how to safely bring them in.  Maybe we make smaller changes every day, like Google does with Chrome, such that they never notice.  Or maybe we ask them to ‘check the box to join our test group and get cool features before everyone else!’  But we must do it, or we will fall behind in giving the users what they want, and giving them a solid, safe, secure product.

    Until then, we’re not analyzing or assessing actual risk, we’re merely players in risk theater.

  • United States Patent: 8010458

    An online social network is provided in which members of the online social network control who may view their personal information and who may communicate with them. The members control who may view their personal information by setting a visibility preference. A member may not view another member’s full personal profile if the measure of relatedness between the two is greater than the visibility preference of the other member. The members also control who may communicate with them by setting a contactability preference. A member may not communicate with another member if the measure of relatedness between the two is greater than the contactability preference of the other member.

    via United States Patent: 8010458.

    It’s official. The US patent office is useless.