Category: How To

Diving Into Varnish
We use it at DreamPress a lot, and I’m still learning its ways, but with me, the best way to learn a thing is to do a thing. So when I had a random server crash with nginxcp, I decided to play around and see about using Varnish on my server instead.

Varnish is an HTTP accelerator designed for content-heavy dynamic web sites (like WordPress). Unlike nginx, there’s no support for SPDY or SSL, which I can’t use anyway unless I spring for another server in front of my Apache box to be a true nginx box. Since I wasn’t getting any benefits out of nginx for those, I’m not too worried about it here yet. Should the world go to SSL, then my POV will change. The Varnish gurus aren’t fans of SPDY as it happens, which I find fascinating.

Back on point. I’m going to use Varnish as a proxy, which means when someone comes to my server to ask for a file, Varnish will first check itself for a cache and then if it’s found, serve it without touching Apache. Apache is slow. This is good! While nginx can handle static files rather well, I found that where I ht slowness people told me to use a CDN. That’s nice, but I don’t want to right now, so it makes nginx less of a draw. On the other hand, Varnish will fill in the gap where Apache + mod_php == poor static-file performance. And yes, I’m using mod_php.

Installing Varnish

First change Apache non-SSL port to 8080. I’m on WHM for this particular box, so I go to WHM -> Server Configurarion -> Tweak Settings and set value of field Apache non-SSL IP/port to 8080

Next I install the Varnish RPM for RedHat REL6. This can be either Varnish 3x or 4x, but I picked the latest version.
```
rpm --nosignature -i https://repo.varnish-cache.org/redhat/varnish-4.0.el6.rpm
yum install varnish
```
Edit the config file – /etc/sysconfig/varnish – and set the VARNISH_LISTEN_PORT to 80.

Now we edit /etc/varnish/default.vcl with what we want.

Deep breath. A whole heckuvalot changed from 3.x to 4.x and it took me a couple hours to bang out, since my examples were all from Varnish 3.x. In the end, I made my own fork of DreamHost’s Varnish VCL. Grab my Varnish VCL Collection and I use the wordpress-example.vcl as my default. It’s a whole ‘nother post on how I did that one. A lot of trial and error.

The default VCL is skewed to WordPress in a specific way: If you’re logged in or have a cookie that isn’t the default WP cookie, or are on SSL, you do not get cached pages. This means my site will be slower for me.

Configuring Your CMS

Speaking of WordPress… Here’s the major difference between it an nginx: I need a plugin for WordPress. I took over Varnish HTTP Purge last year in order to fix it (instead of fork it) for DreamPress, and in doing so I’ve added a lot of little tweaks, like a ‘purge all’ feature and a button on the toolbar.

Oddly, this is the reason I didn’t want to use Varnish. Where nginx just works, needing a plugin means I have to either install and activate for everyone using WordPress or any other CMS on my system, or I have to figure out a way to not need a plugin? Oh, and I don’t just used WordPress. Ugh.

This is moderately trivial to do with Mediawiki but I came up full short when I looked at Zenphoto. While I don’t post often to it (once a week generally), I do post a lot of data and I need the purge to be done. Certainly I could code in a system for it, like I did with WordPress, using a CURL call.

But it’s the need to do that for Varnish that made me make faces.

Not using Varnish

At the end of the day, while I did get Varnish up and running, I chose not to use it. Yet. I have to overcome some hurdles with other apps not knowing how to play well with purging, and figure out how to command purges like I do with WordPress. You can see I have my work cut out for me porting a WordPress plugin to Zenphoto.

In addition, I’m not really sure I like the fact that I have to do that. Certainly I can just let the cache expire on it’s own, but that seems to somewhat defeat the purpose of having it be able to handle dynamism as well as it does if it can’t magically detect when my content changes, and the cache needs a bump.
14 July, 2014
Mailbag: Multisite Files
Another mailbag! This one is a few people… No, this is the one most people ask me. I understand why.

@jbenton @Krogsgard Basically read everything @Ipstenu has written (including when she suggests to not use it).

— Andrew Nacin (@nacin) December 20, 2013

I really do understand it’s intent, it’s goals, and I try to keep up with it. I use it daily. When I write plugins for it, I spend hours trying to decide how to properly support it. I may even write in checks on the ones that don’t work on Multisite to auto-fail when you activate and prevent you from using them. I will argue till the cows come home that Multisite is for multiple separate sites. But that said, there’s also an exception to every single rule.

Let’s get to the letter! Kevin in Canada has a Multisite with a uploaded files conflict:

The site is setup as default /files for the upload dir. Problem is, my client already has a directory in the root called /files. It cannot be changed as it houses software exe’s and needs to remain as /files. I need to set the upload path to /media for the images and not mess the site up. I read your blog post about this issue but wondering if you could clarify the steps. Right now, software download requests are redirected to the homepage!

The steps are, alas, complicated because the situation is complicated. The blog post Kevin’s referring to is called Dumping ms-files. If you’ve installed WordPress as of WP 3.5, this is the new default anyway. That would mean that Kevin has an old Multisite.

The ‘easiest’ option would be to start over with Multisite, use a fresh install of 3.9 and go from there. Second to that? Well the dumping MS files stuff is not easy and it really can’t be easy. You can try trepmal’s directions but after that you’re getting into writing a little bit of your own custom directions, and that is going to be really complicated.

I’m not 100% clear on why these can’t be moved, so I’m guessing that the issue is the lost redirections. While I believe good URLs never change, I think that redirecting them is okay. These are .exe files, which means if, before the .htaccess rules for WordPress, you were to put in a rewrite rule and check ‘If you’re looking for /files/*.exe, go to /otherlocation/filename instead’ that might get around it, or even to say ‘if files and NOT exe.’ Maybe something like this:
```
# uploaded files
RewriteCond $1 !\.(exe)
RewriteRule ^files/(.+) wp-includes/ms-files.php?file=$1 [L]
```
Mind you it’s weird. Either you had Multisite first, which would explain the /files/ for uploads, or you had the EXEs in /files/ first, which doesn’t explain how you got this far into it at all, since 3.5 came out in December 2012, and that would imply you built the Multisite in 2012 and have had this problem for at least 18 months, give or take, which is a long time to have a problem and do nothing about it.

If you started with WP 3.5 or later, you just make a /files/ folder and off you go, no code needed. If you started before that, you can follow the directs to undo MS files and then make a files folder and be done. Make sure not to forget about the .htaccess rules!
4 July, 2014
Genesis Favicons Multistyle

For the longest time I used .htaccess to shunt my favicons to the right place based on the requesting domain. This basic logic worked well, but was starting to get weird and it didn’t always work depending on if I was using pagespeed to trim out leading domains or not, and of course when I tightened security it got weirder.

Now I’ve know you can use custom favicon in Genesis and that’s pretty simple. But … I have to make my life complicated. See I use Multisite and I don’t edit my themes. No, really, this theme? I only edited it to update it. I don’t edit them, I use custom CSS and functions to edit via hooks and filters, the way you’re supposed to.

So how do I override favicons for five sites on a Multisite? Why with an MU Plugin!

#2, shows a specific favicon, and does it for #3 and #4 before defaulting to the main site. I ‘hard code’ in the full URL to make sure I don’t mess it up and there isn’t any crazy cross site shenanigans. Obviously this will get a little long over time, and if I ever stop using Genesis for a site, I’ll instead hook into wp_head() and toss in the link for shortcut icon.

30 June, 2014
Mailbag: Multisite Caching

Dave has a question abtou Multisite and caching!

I asked about caching on multisite setups. I’ve got 10 of these, with between 10-50 sites added to each- and the content is static- never going to change it much. I have a feeling I could benefit from caching, you recommended server side. I could use some help here. I know your site says you’re maxed out time wise- I’m thinking I just need someone to help me with one site, and show me what to do. Or, if they want to set things up on all of them. Do you have any recommendations?

Thanks again for the help.

Dave

Caching! There are three main types: plugin, server, and proxy.

Plugin caching is WP Super Cache, or W3TC, and it’s letting WP make a static cache file on demand. Server caching is APC/Memecached and such, which lets the server make a dynamic cache. Proxy caching is Varnish and the like, which puts a cache before the WordPress server, kind of like a static file on another server.

Which one is best for Multisite with static content? Probably a proxy cache there. Though I’m going to go out on a limb and say if you’re not using WordPress for all it’s useful for, maybe you don’t need it at all. If you have a pretty static site, where you’re not changing content all that much, there’s nothing wrong with static HTML. But if you want to use WordPress (like you’re a WordPress only shop) then you have to consider the options. What is your site doing?

This is actually not a Multisite Question so much as a cache management question. The part that’s multisite specific is who to you let manage what aspect of the cache. I maintain a plugin called Varnish HTTP Purge, which is a pretty basic plugin that checks for common actions, like posting or commenting, and flushes your cache selectively on Varnish when those things happen. One of the Multisite specific issues with that code came up when I added in a new feature: Purge Cache.

That feature lets you press a button to flush the whole cache. Pretty cool, right? The problem was Multisite with Subfolders. See, the way the cache flushes is to say “Dump everything under domain.com!” which works fine with subdomain sites, as that will only flush domain.com and not foo.domain.com. On a subfolder, though, it’ll flush domain.com, domain.com/foo/ and domain.com/bar/ and everything else. That would suck! So the best thought I had was I should lock the flush down to Super Admins only on the main site on a subdomain setup. Normally I allow the site admin to flush their own cache, but this way only the Super Admin could make a full flush.

And that’s the crux of the issue with caching on Multisite. Do you want each admin to maintain their own cache? Do you want it to be something you control for everyone? W3 Total Cache can do that, by the way. You can do it both ways with that plugin, and only control what you want to control for the network. But what about if even I don’t want to do that? What if I want to be lazy and trust my tools? That’s when you look at the server itself and try to make it as brainless as possible. Decisions, not options, remember?

And that brings up another question. Why do I want to manage who flushes at all? In my Varnish example, it’s because careless flushing clears the whole cache, which means my server and site have to recache everything and it slows everyone down. Obviously I want to limit that as much as possible. The other reason is work. People who have a managed hosting site generally aren’t the most technical savvy users in the world. Some are, of course, but many are more likely the ones who don’t want to know about the crazy server stuff. This is great, but it means I want to make their life easier so I have to provide less support, then I want things that do their job and do it well, and that means taking options away and making decisions for them.

How would I cache a static Multisite network and fully control it as a network admin? Probably server side caching like Varnish or ZendAccelerator Memcache. If I can’t do that, I’d use W3TC and lock it down so only I, the super admin, in my fancy cape, can edit settings and purge the cache. And always, always, remember to only catch what should be caught in your cache. If you have a site that’s a store, account for that.

27 June, 2014
The Wheel Is Fine
“I have an idea for a plugin!” he said to me. Everyone does, but I encouraged this question. “I want to make the media embeds in WordPress more responsive.” I paused. There are already a lot of those, like Responsive Video Shortcodes. So I asked what made his different. “I’m going to make NEW shortcodes!”

I see a lot of plugins. Many times I see plugins where people want to tweak a normal part of WordPress, like the gallery or the embeds. The problem I see is that instead of extending those, they decide it’s better to make a plugin that recreates all the shortcodes and embeds, but with a new name, so users have to remember to use (which is default to WordPres) or [youtube] (which is in Jetpack) or [someothervideotool] (which I made up for an example).

Now I’m guilty of this myself! I have a private plugin which is HTML5 video embeds and it gives the video in HTML5 format, which I could do in the default embed code, but it also plunks a link underneath that says “Can’t see the video? Click here.” And the Click Here goes right to the mp4 video. Because I have some rather dim people. But. I don’t have to reinvent the wheel here. I could instead filter that shortcode just to put my little blurb on the end! Doesn’t that seem smarter?

The first time I thought about this, I said “What I need is to filter and slap something on the end!”
```
function halfelf_oembed_filter($html, $url, $attr) {
    $html .= "<p>I am a monkey.</p>";
    return $html;
}
add_filter( 'embed_oembed_html', 'halfelf_oembed_filter', 10, 3 );
add_filter( 'video_embed_html', 'halfelf_oembed_filter', 10, 3 );
```
This puts “I am a monkey” below every embed, which is great for stuff like YouTube and Twitter. The second filter is for videos when you’re using Jetpack. And this works great, I could even wrap the HTML in something, like a div or js code, and enforce responsiveness. You can check the URLs, see if it’s youtube or whatever, and apply different settings per URL. Works nicely. I actually have my just div-wrapping for padding reasons.
```
    $html = "<div style='padding: 5px;'>".$html."</div>";
```
This works brilliantly, however … I’ll get to that in a second. Now, for my friend’s initial idea, once we banged on this, he said he was just going to use the other plugin, but embed-filtering was way easier than trying to over-ride the shortcodes and reinvent the wheel. It also makes it easier to upgrade.

But then there’s my problem with my site and the need for that “Hey, get the MP4 here!” message. You see, 90% of the videos I have on this particular site are locally uploaded, and since I always upload an mp4 and a webm (and sometimes an ogg), how do I parse it?

First of all, I’m using the Video Shortcode, so I know there’s a simple filter for wp_video_shortcode_override (it works roughly the same as img_caption_shortcode but that’s not what I want to use. I don’t want to replace everything, that’s my whole point here. I just want to make a link! In addition, I know I’m only going to want this when I have a video with a defined mp4, so why not just filter wp_video_shortcode instead?

So that’s what I did:
```
function halfelf_video_shortcode($html, $attr) {
	
	if ( !empty( $attr['mp4'] ) )
	{
		$html .= "<p>Can't see the whole video? Click <a href='".$attr&#91;'mp4'&#93;."'>here</a>.</p>";
	}
	
	return $html;
}
add_filter( 'wp_video_shortcode', 'halfelf_video_shortcode', 10, 2);
```
This slaps a link for my tech challenged friends on browsers that refuse to tolerate videos.
23 June, 2014
Local Backups
You heard about CodeSpaces didn’t you?

On June 17th they got hit with a DDoS. It happens. On June 18th, the attacker deleted their data. And the backups. Because the backups were on the same server… You can read the story here and make up your own mind.

But that brings us to this. Are you making your own, personal, backups?

My server makes entire server backups every day and collocates them, but I also have my own backups of my own, personal, data. Not my email. If that blew up today I would lose nothing I can’t get back. That’s right, I don’t keep much email. If it’s important, I store it on my laptop, on iCloud or Dropbox, and I backup my laptop to TimeMachine. Oh and I check that backup regularly.

So how do I backup my sites? It’s three fold.

On the Server

I use a DB script from Daniel D Vork. He backs up files to DropBox, which is cool, but for me, I have this script on my server and it stores to a non-web-accessible folder called ‘backups’:
```
#!/bin/bash
 
USER="your_user"
PASSWORD="your_password"
OUTPUT="/Users/YOURUSERNAME/backups"
 
rm "$OUTPUT/*gz" > /dev/null 2>&amp;1
 
databases=`mysql --user=$USER --password=$PASSWORD -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`
 
for db in $databases; do
    if [[ "$db" != "information_schema" ]] &amp;&amp; [[ "$db" != _* ]] ; then
        echo "Dumping database: $db"
        mysqldump --force --opt --user=$USER --password=$PASSWORD --databases $db > $OUTPUT/`date +%Y%m%d`.$db.sql
        gzip $OUTPUT/`date +%Y%m%d`.$db.sql -f
    fi
done
```
That script is called every day at midnight via a cron job.

Bring it local

On my laptop, under the ~/Sites/ folder, I have a folder for each domain. So there’s one for ipstenu.org (which is where this site lives), and in there are the following:
```
backup-exclude.txt       backup.sh          log.txt
public_html/
```
The public_html folder is a full backup of my site files. It’s not that crazy, don’t panic.

The backup.sh file does an rsync:
```
#!/bin/sh

cd $(dirname $0)

TODAY=$(date)
echo "
-----------------------------------------------------
Date: $TODAY
Host: ipstenu.org hosted sites
-----------------------------------------------------\n" > log.txt

echo "Backup files..." >> log.txt
rsync -aCv --delete --exclude-from 'backup-exclude.txt' -e ssh backups@ipstenu.org:/home/ipstenu/public_html/ public_html > log.txt

echo "\nBackup databases..." >> log.txt
rsync -aCv --delete --exclude-from 'backup-exclude.txt' -e ssh backups@ipstenu.org:/home/ipstenu/backups/ databases >> log.txt

echo "\nEnd Backup. Have a nice day." >> log.txt
```
Backups is not the name of the account but I do have a backup only account for this. The backup-exclude.txt file it calls lists folders like ‘cache’ or ‘mutex’ so I don’t accidentally back them up! It’s simply just each file or folder name that I don’t want to backup on it’s own line. And yes, I like pretty output in my logs so I can read them when I’m having a brainless moment.

The cd $(dirname $0) at the beginning is so that I can call this from other folders. Remember! If your script uses relative paths to access local resources, then your script will break if you call if from another folder. This has a reason why in the next section.

Automate that shit!

I’m on a Mac. I decided I wanted that backup to run every time I logged in to my computer. Not rebooted, logged in. And waking from sleep. That became problematic, but let’s get into this code.

Writing the scripts

First I made a new folder called ~/Development/backups and I’ll be stashing my code there. In there I have a couple files. First is website-backup.sh:
```
#!/bin/sh

/Users/ipstenu/Sites/ipstenu.org/backup.sh
/Users/ipstenu/Sites/othersite.net/backup.sh
```
Basically for every site I want to run backup for, it’s in there. This is why I have the change-directory comment on the backup scripts.

The other file is my launchd file, called com.ipstenu.website-backups.plist and I got this code from stackexchange:
```
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
   <key>Label</key>
   <string>com.ipstenu.website-backups</string>
   <key>Program</key>
   <string>/Users/ipstenu/Development/backups/website-backup.sh</string>
   <key>RunAtLoad</key>
   <true/>
</dict>
</plist>
```
Instead of copying the file, though, I did a symlink:
```
ln -sfv /Users/ipstenu/Development/backups/com.ipstenu.website-backups.plist ~/Library/LaunchAgents
```
This lets me change it if I need to, which I doubt I will. I’ll just edit that .sh script. The filename of the plist is intentional to tell me what the heck it is.

But wait, what about waking from sleep? Logging in from a sleeping computer is not the same as a log in to a Mac, and there’s no built in tool to monitor sleep and wake for some reason. There are apps that can do it, but there’s also SleepWatcher, which can be installed via Brew! Since I’m running an rsync, it’s not a big deal to run multiple times a day. Heck it may actually be faster.

First we install Sleepwatcher:
```
brew install sleepwatcher
```
Now Sleepwatcher looks for user scripts named ~/.sleep and ~/.wakeup which sure makes my life easier. My ~/.wakeup file calls website-backup.sh, and while I could have it repeat the code, I chose not to for a reason. I know my backup scripts will live in ~/Development/backups/ so I can add a new one for something else without messing around with more than one file.

Do you remember launchd a moment ago? We want to use that again to tell Sleepwatcher it’s okay to run on startup or login. This time, since we’re only using Sleepwatcher for sleep and wake, we can symlink the sample files to the proper LauchAgents directories. In my case, it’s only running for me, so it’s all local:
```
ln -sfv /usr/local/Cellar/sleepwatcher/2.2/de.bernhard-baehr.sleepwatcher-20compatibility-localuser.plist ~/Library/LaunchAgents
```
If you’re interested in doing more with sleepwatcher, read Mac OS X: Automating Tasks on Sleep by Kodiak.

Finally we’re going to load both of these commands into launchctl:
```
launchctl load ~/Library/LaunchAgents/com.ipstenu.website-backups.plist
launchctl load ~/Library/LaunchAgents/de.bernhard-baehr.sleepwatcher-20compatibility-localuser.plist
```
Now every time I log in on my laptop, it runs a backup, be that a real login, or a wake-from-sleep one.

And remember, this is on top of my full server backups and my personal git repository for my code, so I have my data backed up in the important places. Everything on my laptop is backed up to the TimeMachine, so really I can just look back a year or three and find that html file I used once.

The other thing I do is check these backups pretty regularly. I scheduled a day every month to check that everything’s working right, that the files are restorable, and that I feel secure. Thus far, the most I’ve lost has been 16 hours of work on a Wiki.
20 June, 2014