Large Files Don’t Move Well

Every time you email a file to yourself so you can pull it up on your friend's laptop, Tim Berners-Lee sheds a single tear.The internet is great for small files. Email, Twitter, Tumblr. When we make webpages, we push to make them lean and mean. So what happens when you have a 200 meg video you need to send out to local news outlets?

The best way is to toss it on a USB drive. I have two in my purse at most times (one’s a novelty R2-D2 my friend sent as a thank you, I keep all my PDFs for software on it). But wait, you say, it’s 2012. Why can’t I use something else?

The problem is that the methods by which we can transfer large files aren’t, generally, user friendly. I mean, that comic there on the left is pretty much why it sucks. Click on the picture to see it bigger. The only think Randall didn’t list was Torrent, which I love for how it handles files, but it’s really not ‘non-nerd’ friendly to set up. To use, it’s easy, but making a torrent, distributing it, and seeding it, takes time.

A friend of mine suggested HFS which stands for HTTP File Server. It lets you treat HTTP as FTP. Just as secure as FTP too. But again, it’s a pain to set up.

So what happens when two simple guys want to share a movie they’re working on? They drive over with an external hard drive and share the data. Because that’s the fastest, safest, best way to do it.

We’re missing something here, aren’t we?

I got to thinking about this when MegaUpload, a filesharing site (mentioned in the comic) was shut down on the 19th for violating copyright. See, some people were uploading material they didn’t have the copyright for to MegaUpload. Why? Because anyone could sign up and upload anything. Of course.

I meant MEGAUPLOAD in that tweet, but Eric’s reply got me thinking more.

How much illegal material goes through the US postal service? I mean, let’s say I download illegal software, put it on a USB drive, and mail it across state lines to my friends in Texas. How many laws did I break? A lot! The US Postal Service sends a lot of really weird mail but they do have rules. The trick is getting caught. They don’t check DVDs, books, or USB drives because there’s no need to.

Going back to MegaUpload, they were shut down because someone used them to transport illegal items. If we apply that mentality to physical mail, then we’re talking about shutting down the entire postal service because Bobby Dumbass mailed his brother a video of himself jerking off. Oh yes, that’s illegal.

This isn’t unnecessary hyperbole.

The reason MegaUpload is so popular is that it made it fairly easy for the laymen of the world to upload large files. The problem with it is that it’s filled with distractionary popups and the like. I’m using ‘is’ since I think it’ll be back. Still, MegaUpload filled a niche that is desperately needed. How do the tech-newbies upload large files? I’m capable of making and seeding a Torrent, but those aren’t easy and rely on other people seeding to speed it up, plus a level of tech-savvy from the receivingt end where someone knows how to use a torrent file and pull the data back down.

On the other hand, when I download something from Apple, say a 300meg update for my iPhone, I’m downloading … a 300meg update. Apple has a gazillion severs in the US. Why not use Torrent technology to let me pull down the file in chunks, instead of in order? Is this because of how Torrents work, or are they just scared because of what happened to other Torrent sites?

Torrents (bittorrents) are amazingly impressive to me. In regular file downloads, you pull a file down ‘in order’, essentially, like a printer. One line at a time is sent, downloaded, rendered and output. Torrents spin that around, and work by downloading small bits of files from many different web sources at the same time. It’s like watching the movie Memento. The story is told out of order, but in the end it makes sense. That means if I’m downloading one bit from Joe in Arizona and another from Dan in Nebraska, I still get the same file, and I get it faster because if I lose connection to Dan, my Torrent app finds Barbara in Iowa who’s also seeding the file and I keep downloading.

Today, if I lose connection in an FTP download, I have to start over. A torrent I can stop and start all I want. I know, that sounds totally awesome, right? So why aren’t we using it more?

I don’t know. I think because it’s seen as ‘dangerous’ by the copyright moguls. It makes it too hard to track infringement, and makes everyone culpable. Which it really doesn’t. I mean, I guess they’re just afraid and don’t understand that this power is already there, and that it’s always been there. These are the same people who wanted the VCR to die an ugly death because it would ‘hurt sales.’ Protip: It didn’t. Neither will this.

So let’s push our tech companies to come up with a better way to share large files, a way the non-tech people can use. Make it easy to set up on your computer, make it easy to understand, like email and the basics of the web. Make it fast to upload, faster to download, and easy to link to. Make it easy to keep private if we want, or public if we don’t. Make it simple to report copy infringement too, and use it as a legal way to send large files, like movies, so the Hollywood people can give us a viable, workable, alternative to theft.

We have the tools, let’s do it!

About these ads
StudioPress Theme of the Month

Comments

  1. Have you tried WeTransfer? Great for transferring large files between one or a few people. It also sounds like you might enjoy using box.net.

  2. Just as secure as FTP too

    So… not very?

  3. Oddly enough, we just had this exact problem where I work. We frequently have to download/upload large files for our clients to convert their customer data into new formats. Anywhere between 50MB to 4GB at a time. Until last week, we used FTP … and if the connection was reset while transferring the file, we just had to restart.

    Instead, I built a chunking system loosely based on torrent technology. I cut up the file into smaller pieces and upload them (in order for now) separately. The connection can be reset, but restarts exactly where it left off. No more retrying an upload of a 500MB SQL dump that breaks halfway through.

    Oddly enough, Microsoft actually uses torrents for some of their file downloads. If you ever take a really close look at how their download manager works … it’s just a very well-branded torrent client. I noticed this when downloading the latest service pack for Visual Studio.

  4. Few points:

    HTTP 1.1 supports resuming file downloads just fine. Just make sure your server is doing HTTP 1.1 and not 1.0 and you should have no problem with pausing/resuming file downloads using modern browsers.

    FTP supports resume too. The Filezilla client can handle it just fine, as long as you have a decent FTP server that supports it as well.

    Many systems are know to use torrent-like transfers internally. Steam uses a system very similar to BitTorrent, except that it only downloads from Steam’s content servers, not from other peers. World of Warcraft uses the Blizzard Downloader, which used the BitTorrent protocol specifically in early versions. Unknown if it’s still using that though.

    The problem isn’t in the protocols. The problem is in the interfaces. Transferring large amounts of data is largely a solved problem, but it’s not a solved problem for users who only know how to do things like email and such. People who don’t understand the concept of a “server” to begin with can’t overcome that gap easily. Thus the need for third party transfer services.

    • Yes, thank you. The interfaces.

      Though honestly, the protocol Apple uses makes me brick-shot angry, since I’ve had it crap out on me too many times. With two iphones and an iPad, upgrading all three on a dodgy network was a nightmare. Now that I’m on higher high-speed, it’s great, but I desperately wanted torrent-esque streaming.

      We have an ‘app’ for email, though so many people use webmail now, I fear that’s where the next ‘FTP’ will be.

      Then again, we could still use Archie…

  5. Another favorite option is File Apartment (http://www.fileapartment.com). Easy to use, no software to download or registration, up to 1 GB, safe, secure, and free customer support.

Half-Elf? Try Half OFF WordPress ebooks!