Author: Ipstenu (Mika Epstein)

The Fiscal Responsibility of 25%

“We’re going to deploy on December 20th.”

I winced.

I’m not Christian and December 25th is just another day when I binge watch Netflix. The 24th? Movie night! But I picked up a habit at the Bank and that was not deploying code for the last two weeks and first two weeks of the year unless the company was going to be fined if we didn’t.

I was brutal about that to people. I was harsh and mean and demanded lengthy justifications. I made them speak to senior management, who were hard to find around that time of year because every one was on vacation.

The reason for this was that the bank had a clear cut fiscal responsibility not to go down between Thanksgiving and a bit after New Years. That was when year end processing happened, and that was when many companies who used us were processing the most orders. For a lot of people, business booms right when everyone wants to take a break to be with their family.

WordPress runs 25% of the Internet.

We use it for blogging, for building Facebook-eqsue sites, for running ecommerce stores.

That means and we the developers of WordPress now have a responsibility not to break when we upgrade people. A huge responsibility. And it’s one we can never give with 100% assurance because of one simple fact.

We made WordPress open.

Anyone can make a theme or plugin. And while we do our best to test with core WordPress, we cannot test all of the 45k plugins in the repository yet. The best is maybe we could write a script to check for fatal errors on activation. But even then, can we test all 45,000 against all possible permutations of combinations?

That’s an incredibly massive number. All my factorial calculators, even Google, just said ‘Infinity.’ And we add about 9000 plugins a year. This is staggeringly huge and it gets bigger every year

But with this increase in share and use comes an incredible responsibility to 25% of the web. We cannot break their sites.

Of course I know that’s impossible. There will always be outliers. And even with the large user base that companies like Yoast have, the dearth of willing and capable Beta Testers for a free product is going to bite us. It’s part of what I asked what I did at the Town Hall at WCUS — Are we going too fast?

Speed cannot exclude us from a responsibility to our users. And with the increasing provenance of online stories and websites for everyone, pushing a change when we know that the majority of the world is celebrating something between Nov 15th and January 15th is reckless. Look at how many people want time off in those months to be with family. Look at how many businesses are running sales. Look at the amount of data transfer that spikes.

And then picture what happens when an update has a small bug that takes down one site in a thousand. 1/1000 of 1/4th of the entire Internet. If that didn’t make you shiver, do the math again. Imagine if Apple went down because they pushed an update right around Christmas?

The answers change sometimes, though. What if it was a security fix? Would that change your mind? It would change mine. A major upgrade around Christmas worries me. A minor one, not so much.

It may be time to call a year end moratorium on updates to our systems and apps. If they’re business and mission critical, test them as best you can, but consider if you have to update before that Christmas rush. Make people jump through hoops to prove they need the new shiny right now. If you know you’re understaffed or under heavy load, consider that as much as anything else.

1 January, 2016
Looping Hugo
I’m ending the year with something non-WordPress. But it has a weird JSON related journey.

I asked three questions at the end of my last post:
1. How do I properly mimic a for-loop with range?
2. Can I make a responsive gallery out of Hugo?
3. Can I power a Hugo site with WordPress JSON?
This is the answer to the first one. However in the solving, I made a breakthrough in my head about how one calls data from JSON and structures it.

But I’m getting ahead of myself, becuase changing a for loop from Jekyll to Hugo was harder than I expected. Way harder.

One of the things I check in some posts is what the rating is. I set a parameter called ‘rating’ and if it’s there, I ran a quick check to determine how many stars to show:
```
<strong>Rating:</strong> {% if page.rating %}{% for i in (1..page.rating) %}<i style="color:gold;" class="fa fa-star" name="star"></i>{% endfor %}{% if 5 > page.rating %}{% assign greystar = 5 | minus: page.rating %}{% for i in (1..greystar) %}<i style="color:grey;" class="fa fa-star" name="star"></i>{% endfor %}{% endif %}{% else %}<em>Not Available</em>{% endif %}
```
I was pretty damn proud of that loop. Check the rating, output a gold star for each number over 0 and a grey star for every number between the rating and 5. It’s simple but it’s quick.

Transposing from Liquid to GoLang was not as hard as all that. {% %} became {{ }} and {% endfor %} became {{ end }} and, at first, that was the easy stuff. The majority of my logic was a one-to-one translation. Change page.rating to .Params.rating and so on and so forth.

The order of the if’s was strange to me, in that it became {{ if eq A B }} (except when it wasn’t) and I was used to thinking {% if A == B %} — it was fairly easy to overcome. In short order, all my simple if-checks were done.

But those damn loops! I had the ugliest “if A == 1, show 1 gold and 4 grey” configuration. And worse, I had cases where I was checking “If there’s an entry in the Filmography for this show, get the rating from that and not the page itself.” The Filmography file was a straight-forward JSON file, you see (told you JSON was involved).

Back to the first problem. I knew if I wanted a shortcode to say “Get me a list of all pages, by date, where section is news” I did this:

{{ range where $.Page.Site.Pages.ByDate "Section" "news" }}

And that $.Page.Site.Pages.ByDate call changed depending on what template I was on and what I was calling. For a shortcode I had to pass though page to site to pages. On a template I could do .Data.Pages.ByTitle (no $ needed either). I’m still at the trial and error stage of figuring out which call and where, but I know I’ll get there. It’s just mastering new syntax and understanding where I’m calling from and what variables are available to me.

That’s really okay. It took me years to visualize the interrelations with WordPress themes and functions and plugins, and even then sometimes I’ll output an array in ugly, unformatted ways just to read through and make sure I understand what I’ve got at my disposal. This is normal for most of us.

And it was fairly clear that if I wanted a shortcode to call the data file and not a list of Section pages, there was code for that: $.Page.Site.Data.filmography (back to Filmography again). And that worked fine. Right up until I wanted to say “For all values where X equals Foo…” and I was back to my loop-headache.

Now. GoLang does have a for loop!
```
func main() {
    sum := 0
    for i := 0; i < 10; i++ {
        sum += i
    }
    fmt.Println(sum)
}
```
And I though that I could just use that. Nope! So I asked myself why did {{ for i in (1..Params.rating) }} fail?

First of all, that ‘in’ should be := instead. But even so, when I ran it I got “ERROR: function “for” not defined.” That means there is no for function. And no loop. After looking at how I was iterating in other places, I did this:
```
{{ range $index, $element := .Params.rating }} 
   ONE
{{ end }}
```
ERROR: 2015/12/12 template: theme/partials/rating.html:3:26: executing “theme/partials/rating.html” at <.Params.rating>: range can’t iterate over 3 in theme/partials/rating.html

The 3, in that case, had to do with a rating of 3. This makes a little sense, since you’re supposed to use range to iterate over a map, array or slice. A number is none of those.

Thankfully I found a ticket talking about adding a loop function to Hugo that was addressing what I was trying to do and finally I had an answer:
```
    {{ range $star, seq .Params.rating }}
        <i style="color:gold;" class="fa fa-star" name="star"></i>
    {{ end }}

    {{ $greystar := sub 5 .Params.rating }}

    {{ range $star, seq $greystar }}
        <i style="color:grey;" class="fa fa-star" name="star"></i>
    {{ end }}
```
The idea there is that seq makes a sequence of the value of .Params.rating for me. If the value is 3, I get an array of [1,2,3] to work with. And that’s something range can itterate over!

Except…

at <sub 5 .Params.rating>: error calling sub: Can’t apply the operator to the values in theme/partials/rating.html

Now the amusing thing here is that the code worked! Even if it wasn’t seeing .Params.rating as an integer, it did the math. I suspect it’s related to this old bug, where variables from front matter are strings and not integers. Except that he said it worked in YAML and that’s what I’m using.

This was the most perplexing issue. How is a number not a number if it’s being numbered? And then I noticed that I was able to make a sequence, so clearly at that point it knew it was a number. And then I did this:
```
    {{ $goldstar := seq .Params.rating }}
    {{ $goldstar := len $goldstar }}
```
And yes it works.

I’m basically saying “Hey, how many ones are in the number X?” and saving that as a number. There’s no real reason I can comprehend why it failed to work, but there it is. I’ll file a bug as soon as I can figure out how to explain that in a way that doesn’t make me sound crazy.

How did all this teach me about JSON? You’ll have to wait a few days.
30 December, 2015
Porting Jekyll to Hugo
Importing content is always the bane of moving systems.

Now one of the things Hugo has is a cool command called $ hugo import jekyll – and yes, it does exactly what you think it does. This was interesting to me since I wondered if I could switch my library from Jekyll to Hugo and yes, yes I can.

The site I have on Jekyll has a few more posts than the first one I built on Hugo. This one has around 1500 pages. Excited, I ran the command:
```
Importing...
Congratulations! 0 posts imported!
```
That’s not right. I spent a few hours trying to run it in various permutations, but I was never able to get it to run right. Thankfully, I didn’t need to since I could just copy my content over. There was no translation needed, as I was already using yaml headers in my .md files so that actually was a pretty painless transition.

The theme on the other hand… Well. I had a complex site with data files and weird junk being thrown around. That was just learning how to translate for-loops for ranges, with some where clauses tossed in.

Was it hard? Yes. But that’s not what you’re really asking.

Was it worth it?

Excuse me while I whip this out:
```
Change detected, rebuilding site
2015-12-11 21:42 -0800
0 draft content
0 future content
1479 pages created
0 paginator pages created
16 categories created
66 tags created
in 4611 ms
```
Yes. The site builds faster, which is a win. It also creates tags and categories that can automatically cross link. I can make ‘sections’ without having to define them in my config file, just by making folders. I can spin up templates quickly. I have a little more redundancy than I’d like, but as I master GoLang, this becomes less and less.

Build time ranges from 11737 ms to 2116 ms. Milliseconds. They were into the minutes with Jekyll, even though Jekyll 3 was notably faster than version 2. It was still taking me about 2 minutes to rebuild the Jekyll site, even with incremental builds working. Plus I had to build and rsync every time, in addition to Git pushing. Now I just Git and done.

Oh and JSON works remotely out of the box. I just have to get better at my range calls now.

What’s Different?

Making a ‘static front page’ with Hugo is harder. In fact there’s still an open trac ticket on it. I was able to use an idea I already had, a custom post param called “type” for tagging ‘index’ posts in the news pages so I didn’t get a list of my indexes when I wanted to list all news from 2002. I leveraged that and tagged my primary index ‘mainindex’ and put a quick check in my index.html in my theme:
```
{{ range .Data.Pages }}
    {{if eq .Type "mainindex" }} 
        {{ partial "content.html" . }}
    {{end}}
{{ end }}
```
Instead of doing a normal loop where, like a blog it shows me posts in reverse order, it just showed that specific content.

But…

If there’s a file in that Section with the type of ‘Index’ then it shows that. Of course… While index.md doesn’t get used in the main content folder, it does in Sections. If I have a file named index.md in a section, that makes index.html like I expected. Since I still wanted lists in the majority of cases, I tweaked the content of my default list file (list.html) so that I could always show a ‘main’ file:
```
    {{ range .Data.Pages.ByTitle }}
        {{ if eq .Type "index" }}
            {{ .Content }}
        {{ end }}
    {{ end }}

    <ul>
    {{ range .Data.Pages.ByTitle }}
        {{ if ne .Type "index" }}
            <li><a href="{{ .Permalink }}">{{ .Title }}</a></li>
        {{ end }}
    {{ end }}
    </ul>
```
Then I renamed all the index.md files in the Sections to main.md but this created another issue. Now I had two URLs that worked. I could have put this into the template for the section to avoid this, but I want to separate content and theme as much as humanly possible. WordPress habits. For now, I’ve left this alone. If someone is clever enough to go to a non-linked, non-sitemaped page instead of the index, I think my SEO will live. It’s more than I wanted to play around with loops.

Speaking of the loops and lists, getting a list of a ‘sub section’ is not currently possible either. Rather, they do work, but there’s no default templates for sub-sections. This was a small problem since I structured my site like this:
```
.
└── content
    ├── news
    |   ├── index.md          // /news/
    |   ├── 2001
    |   |   ├── loch-ness.md  // /news/2001/loch-ness/
    |   |   └── monster.md    // /news/2001/monster/
    |   └── 2002
    |       ├── nessie.md     // /news/2002/nessie/
    |       └── spock.md      // /news/2002/spock/
    └── posts
```
Now, I don’t need a custom template for each sub-section, but I did want to use my custom list template for each sub-section. To work around this, I made a template in /layout/section/news.html that had this:
```
	{{ range .Data.Pages.ByTitle }}
		{{ if and (eq .Type "index") (ne .Params.topic "index") }}
			{{ .Content }}
		{{ end }}
	{{ end }}
```
I know this is weird. The Type of index means “I am the main index file of the section” while the topic index means “I am an indexing file and not a normal news article. This gives me my main index of /news/ giving me a custom content page with information like I wanted, using the main.md format I use in all sections.

Then in each sub-section, I use a very simple index.md file:
```
---
title: News Articles (1992)
author: Mika Epstein
layout: news
topic: index
date: 1992-01-01
permalink: /news/1992/
categories: ["News"]
tags: ["1992"]
---

{{< news >}}
```
That weird bit in the content is a shortcode ala Hugo, which calls the file /layouts/shortcode/news.html that has this:
```
{{ $date := $.Page.Date.Format "2006" }}

<table class="infobox">

    <thead><tr>
        <th>Date</th>
        <th>Source</th>
    </tr></thead>

    <tbody>
    {{ range where $.Page.Site.Pages.ByDate "Section" "news" }}
        {{ $thisdate := .Date.Format "2006" }}

        {{ if and (eq $date $thisdate) (ne .Params.topic "index") }}

            <tr>
                <td>{{ .Date.Format "01 January" }}</td>
                <td><a href="{{ .Permalink }}">{{ .Title }}</a></td>
            </tr>

        {{ end }}
    {{ end }}
    </tbody>
</table>
```
That spits out a table of all the pages in that section by date.

What’s Next?

It took a lot to get here, or so it may seem. This took me two nights to work out. That’s it. So now I have to figure out the next steps that have vexed me:
1. How do I properly mimic a for-loop with range?
2. Can I make a responsive gallery out of Hugo?
3. Can I power a Hugo site with WordPress JSON?
And that’s a whole ‘nother story.
28 December, 2015
Mailbag: Hugo from iPads

So I’ve done the whole Hugo thing and it’s great and it totally works for me. That 1% itch left from Jekyll is gone. So queue the inevitable…

But what about mobile posting?!

Why on earth the planet is obsessed with posting everything from their phone, I don’t know. Things like Instagram and Twitter make it easy for us to ‘communicate’ (and I use that loosely) and post photos of our lunch. And yes, the iOS app for the iPhone means I can ‘live blog’ but, to be honest, I hate it on my iPhone.

My iPad though… I love posting from that.

And yes, yes I can post to my Hugo run site from my iPhone or iPad. Remember, I can push my new content by running a git push command. The server will build, sync, and clean up on its own. All I need to do it is a Git app on my iPad that doesn’t suck

Working Copy does not suck.

It’s an iOS app that pulls my git repository to my iPad (the whole thing, so make sure you have room). Then I can edit files, commit them, and push. Hugo on the server does the rest, just like it would from my desktop.

Working Copy is different from many other Git related apps because can hook into any git repository. I’m not using GitHub for this project. I want to do 100% self hosting, and that means no GitHub. Also no BitBucket. These tools are fine for what they are, and in fact I pay GitHub for some private repos. But I wanted my repo on my server, in part so I could do exactly what I’m doing.

The tool is incredibly simple. It’s a familiar file navigator, tap through the folders. Tap edit to edit, make changes to the post, hit done. There’s even a preview feature that mostly works. Some of my Markdown files are very weird.

If I wanted to edit in Byword, which is what I do most of my writing in for non-novel related things, I can share and go back and forth. I’d love it if the WordPress iOS app (or Calypso) did that. Write in Byword, send to other app. But even a copy/paste is simple enough. In this case, I can write in Byword and share to Working Copy. The interface takes the title of my document and let’s me pick the folder.

Once I’m done with my edits, I commit my changes. Then I can push, or not. There’s an option to commit+push, but it crashed my app. A skim of reviews showed this happened to others. The iOS interface can be a bit tetchy so having this be two steps doesn’t bother me.

I did find it odd that there was no button to push. I had to go back to the main folder, swipe-left, and press the orange push button. But that, and the weird crash, were it for annoyances.

While free to download, it’s $9.99 for unlimited deployments. It was free for 3 weeks, which I tried out over WordCamp US and found simple and perfect for my workflow.

25 December, 2015
Consenting to Collection

Collecting information on users is something every program wants to do. Doing it is easier said than done. (more…)

23 December, 2015
Disclosure

One of the myriad reasons I push back on WordPress plugins is because someone didn’t disclose enough information about ‘phoning home.’ Phoning Home is simple concept that comes down to this “Don’t send data from the plugin users back to yourself (or anyone else), unless you’re a service. And if you are a service, make this clear in the readme.”

Simple, right? It’s totally easy to understand that we want you to tell people “Hey, if you use this plugin, it will report back to my servers…” Much of the time, this is obvious. A Facebook connection plugin, logically, contacts Facebook. Embedding YouTube playlists contacts YouTube. Those sorts of things we don’t worry about, though if you have your plugin say “This will pull data from YouTube” then it’s better. Sometimes this is less obvious, like if you use geoplugin.net to determine locations. Your plugin probably has nothing to do with that domain, but you still have to tell people about it so they know.

But why do they need to know?

First of all, this is the basic email I’ll send you if I see you’re not explaining the phone-home in your plugin:

Plugins that send data to other servers, call js from other servers, and/or require passwords and APIs to function are required to have a full and complete Readme so we can make sure you’re providing the users with all the information they need before they install your plugin. Our goal with this is to make sure everyone knows what they’re installing and what they need to do before they install it. No surprises.

This is especially important if your plugin is making calls back to your own servers. For the most part, we do not permit offloading of images or code, however in the case where you are providing a service (like Disqus or Akismet or Twitter), we permit it. The catch is you have to actually explain this to the layman in your read me, so they know where data is going.

Clearly there are some basic reasons, like we should know where our data is going for our own safety. There are also some surprising reasons to people who don’t think about these things, like legal ones. You’re calling out to other servers? What if my company legally can’t do business with them? Then we have the tin-foil hat reasons, like I don’t want to do business with Google so I don’t want to have Google JS in my plugin.

All that sounds pretty basic. And some of it is super obvious. If you’re making an app to communicate with Facebook, then it’s logically going to send data to Facebook. None of that surprises anyone, nor should it. With a service, one simply has to be up front about what the product does, what services it connects to, and why.

“This plugin pushes your comments from your Facebook page to your blog, matching users by email addresses with their Facebook accounts (if found).”

I just made that up. But it’s upfront, it’s honest, it’s direct, and it’s clear what’s being sent and where and why.

There’s another aspect to this, however, something that is far trickier and more complex. What happens when your existing tool adds a service?

The obvious answer is that you need to disclose this change to our users. As long as the users know what they’re getting into, then you are golden. The complex answer, the one I can’t really tell you a one perfect answer for, is how you might do this.

Why is this hard? It’s hard because there is no one right way to tell users about the change. There is no one perfect way to make sure users read the information. There is no one way to get all the data to all the people who need to know about the information.

And there sure as hell isn’t one way to make sure no one will complain about any of that.

When you make a change to the paradigm of what your tool does, taking a stand alone tool and adding in a service, you have to consider that a percentage of your users will rebel. This is simply because all those wonderful things you do about disclosing the service for new users have to be transitioned in a meaningful and logical way to existing users. And there is just no way to do that perfectly.

This doesn’t mean you shouldn’t try. This means you have to be creative and innovation and a little ‘in your face’ about the change. You have to give users an option, before the service kicks in, to say if they want their data shared.

21 December, 2015