Categories
How It Is

The Fiscal Responsibility of 25%

Do we have a fiscal responsibility to not update or must we update, regardless of the big picture?

“We’re going to deploy on December 20th.”

I winced.

I’m not Christian and December 25th is just another day when I binge watch Netflix. The 24th? Movie night! But I picked up a habit at the Bank and that was not deploying code for the last two weeks and first two weeks of the year unless the company was going to be fined if we didn’t.

I was brutal about that to people. I was harsh and mean and demanded lengthy justifications. I made them speak to senior management, who were hard to find around that time of year because every one was on vacation.

The reason for this was that the bank had a clear cut fiscal responsibility not to go down between Thanksgiving and a bit after New Years. That was when year end processing happened, and that was when many companies who used us were processing the most orders. For a lot of people, business booms right when everyone wants to take a break to be with their family.

WordPress runs 25% of the Internet.

We use it for blogging, for building Facebook-eqsue sites, for running ecommerce stores.

That means and we the developers of WordPress now have a responsibility not to break when we upgrade people. A huge responsibility. And it’s one we can never give with 100% assurance because of one simple fact.

We made WordPress open.

Anyone can make a theme or plugin. And while we do our best to test with core WordPress, we cannot test all of the 45k plugins in the repository yet. The best is maybe we could write a script to check for fatal errors on activation. But even then, can we test all 45,000 against all possible permutations of combinations?

That’s an incredibly massive number. All my factorial calculators, even Google, just said ‘Infinity.’ And we add about 9000 plugins a year. This is staggeringly huge and it gets bigger every year

But with this increase in share and use comes an incredible responsibility to 25% of the web. We cannot break their sites.

Of course I know that’s impossible. There will always be outliers. And even with the large user base that companies like Yoast have, the dearth of willing and capable Beta Testers for a free product is going to bite us. It’s part of what I asked what I did at the Town Hall at WCUS — Are we going too fast?

Speed cannot exclude us from a responsibility to our users. And with the increasing provenance of online stories and websites for everyone, pushing a change when we know that the majority of the world is celebrating something between Nov 15th and January 15th is reckless. Look at how many people want time off in those months to be with family. Look at how many businesses are running sales. Look at the amount of data transfer that spikes.

And then picture what happens when an update has a small bug that takes down one site in a thousand. 1/1000 of 1/4th of the entire Internet. If that didn’t make you shiver, do the math again. Imagine if Apple went down because they pushed an update right around Christmas?

The answers change sometimes, though. What if it was a security fix? Would that change your mind? It would change mine. A major upgrade around Christmas worries me. A minor one, not so much.

It may be time to call a year end moratorium on updates to our systems and apps. If they’re business and mission critical, test them as best you can, but consider if you have to update before that Christmas rush. Make people jump through hoops to prove they need the new shiny right now. If you know you’re understaffed or under heavy load, consider that as much as anything else.

14 replies on “The Fiscal Responsibility of 25%”

Yes! Please! No major updates (not just WP core– I’m looking at you plugin developers!) between mid-December & mid-January!

Not just e-commerce sites gear up in this time frame. Brick & mortar stores need to announce their specials, and some of my non-profit clients are pushing their annual membership renewals (yes, some organizations persist in using calendar-year memberships ).

@Bet Hannon: releasing updates is fine at any time of the year. If anything, I like getting updates from major software products just before (other people’s) holidays because I can quietly test them without any pressure to install.

The problem is people installing those updates, directly into a production environment. And not even testing them properly!

@Ross McKay: actually the problem is also that we constantly tell them ‘update as soon as possible.’ We are encouraging the behavior of upgrading fast, and in the case of WordPress, we bend over backwards to ensure that minor updates are as safe as possible.

Major updates? Plugins? Not so much. And there in lies a big part of the problem. We cannot simultaneously say to upgrade asap while releasing updates when we know the majority of users don’t have the spoons.

The worst case I saw last year, and leading up to xmas and its critical sales period, involved WordPress rolling out a change to ca-bundle.crt which broke many ecommerce stores:

https://core.trac.wordpress.org/ticket/34935

That ticket has been resolved, and the fix will come out in 4.4.1 sometime “real soon”. There’s also a couple of workarounds (replace ca-bundle.crt, or downgrade to WP4.3.1). However, it highlights the risks of upgrading at any time of the year, but especially heading into a holiday season because 4.4.1 could not come out sooner due to holidays.

Anyone heading into their busiest sales period would have been crazy to upgrade their shop to WordPress 4.4, and sadly many were, to their detriment. Even if they had some need to upgrade to 4.4, testing in a staging site (on the same server!) would have caught that problem though.

So that means some site owners directly upgraded their source-of-income websites heading into the holiday season, without testing first to see if there would be a problem. Ouch.

get a staging environment set up, and use it!
don’t upgrade what ain’t broke just before your busiest sales period, or holidays, or BOTH.

@Ross McKay: The problem is the speed with which the WP community is demanding everyone update. Do it fast. Do it right now. Test quickly and then roll out.

I’m willing to bet that even with a staging environment, that crt issue would still have impacted people. Even when you do everything right, pushing updates and making it socially reprehensible to not update the day 4.4 came out, is causing problems. Hell, people should have tested the rcs! They don’t. The plugin and theme developers don’t.

Perfect world environments only exist when you lock down everything. And even then, WP.com has taken itself down.

And with that in mind, do we not owe it to our users to consider the timing of these things and not drop something right before when we know the people who need to do the testing will be short staffed?

You can’t predict everything of course. But you damn well know that year end means sales and taxes, pretty much universally. You know when Tax Day is in the US. You know when DST happens.

Don’t release when you’re putting people behind an 8-ball.

@Ipstenu (Mika Epstein): right, I see where you’re going there. Yes, the WP community attitude to updates is very bad. It’s like a demand that you keep up, or we’re not even going to help you, and you’re being a fool to yourself and a burden to others if you don’t upgrade.

That attitude really needs to change.

That doesn’t diminish the responsibility that ecommerce sites have to protect themselves from upgrade risk by setting up a staging site and actually testing in it. Far too many people are still just clicking that Update button without even looking to see if there are reported problems, let alone testing in a staging environment. And this with lots of reputable sources publicising the need to do that.

You’re right that the push for releases near critical dates is bad, and the attitude of “update or we disown you” is even worse, but I still want to see frequent updates. I just want to test them somewhere safe first, and I want everyone else to do that too.

I reckon a staging environment on the same server would have caught that .crt problem for most sites too. At least, in most cases, where the payment gateway sandbox had the same CA chain as the live gateway (which would be the vast majority of them I reckon).

@Ross McKay: Sadly a same server environment for staging is considered high risk in many places. An ‘identical as possible’ server is the goal in corporate America, but the crt thing requires specific testing to reproduce.

I’ve seen things like it slip by at the Bank 😐

I’ll chime in on this. I got 3 sites with a client, I did my due diligence and updated two of them soon after 4.4 dropped. A couple days before Christmas, I get an email from the client that he couldn’t get to some pages. It was very confusing.

I dig and yeah, it’s something 4.4. I roll back to 4.3.1.

I get another email a few days later about the other site, same problem. I digged into trac. Found this. 35031

In there was a patch, I applied it. Seemed to work. Didn’t have to roll back.

The 3rd site is ecommerce, I updated it all and tested everything in early November. Since then, I haven’t touched it. I’ve updated nothing. That site is mission critical in a stressful time for my client. If we know everything works, why change it during the most important time of year?

My question, is that if I was on a managed host, what recourse would I have had these scenarios? I’m assuming the managed host would have auto-updated the sites to 4.4. The problems would have been found and then what? Obviosuly, I’d put in tickets to their support. But what really would have been done. I wouldn’t have been able to apply patches to code found in trac and I doubt I could have gotten anyone to roll the sites back to 4.3.1.

From now on, regardless of how nice and new and shiny the updates are to core, I’m going to install x.x.1 updates and wait.

Jonathan

In Dutch there’s this saying; “niet vloeken in de kerk” which roughly translates as “don’t swear in church”. But maybe consider this a confession rather then swearing?

For the latest release of Autoptimize I purposely decided to release the new version (a major update) on the first Saturday between Christmas & New Years.

My reasoning behind this;
(0. the update had been tested extensively by a significant amount of testers, power-users and translators and the release had been communicated well in advance)
1. given the time-of-year and the fact is was a Saturday, less people were bound to install the update on day 0, limiting the damage in case a major bug would still be in there.
2. my day-time job is less demanding the last 2 weeks of the year, so I could spend more time then usual following up on support-requests and bug-reports and release an update (or even rollback to the previous version) if needed on day0 or day1.

All-in-all the roll-out is going pretty smoothly with 26K downloads to date (22% of active sites are on 2.0 already), 7 people confirming it works, 3 extra 5-star ratings and no big issues yet. So guess there’s something to say against having a moratorium? 😉

frank

@frank goossens: Autoiptimize has a smaller user base, though, and slightly fewer variables.

Pretend you’re Yoast. And pretend the plugin is Yoast SEO. And I suspect you’d rethink those dates.

I love Yoast’s products and personally recommend them. His last round of updates were a nightmare due to things he can’t possibly predict, and sadly prove the point. It doesn’t mean his code shouldn’t be used. In fact, the speed with which he fixed it is a testament to his work and his team’s. But the issues that happened are hard to guess because the percentage of his users who do beta test are far, far smaller than wp’s own.

The user base indeed is “slightly” smaller 😉

But my main reason for releasing at that exact point stands and could be valid for all major releases of any plugin: the goal was to limit the impact in case of a major bug by avoiding the majority of my user base upgrading on the same day.

Now wouldn’t it be better (and I’m just thinking out loud here) if wordpress/ wordpress.org would allow “throttled” releases, where a non-security release could optionally get pushed out gradually (e.g. 5% of userbase on day0 +10% on day1 +15% on day2 +30% on day3 + rest on day4)? This would off course require api.wordpress.org/plugins/update-check/ to “lie” about the availability of updates in case of a non-security release that indicates it wants to be “throttled” (both could be flags in the readme.txt’s header)?

@frank goossens: That would sadly rely on the second least reliable aspect of WordPress: Developers who understand what minor releases should be.

(The least reliable are users, but I think that their lack of understanding is our fault as devs and designers for failing to explain properly and make intutive enough)

Maybe, but for developers who do not understand, nothing would change. For developers who do understand, adding that simple, optional “Throttled release” flag in the readme.txt header would allow them to limit the damage a bug could do to the percentage of installations the update is shown to on a given day after the release. I’m pretty sure Yoasts latest releases would have benefited form such an “throttled release” approach 😉

(the “security fix”-flag would be another optional flag, automatically disqualifying throttled releases, but it could serve additional purposes, e.g. indication a release as “security fix” on the update dashboard).

@frank goossens: I’m really glad that you did release then, because v2 fixed some problems with a new theme + WooCommerce for me and I was trying to push that new theme out right about then! 🙂

I think it’s fair to say that your userbase is a bit more technically minded than, say, Yoast SEO’s though…

Comments are closed.