In a nutshell, the paradox is this:
The more likely a person is to test Beta and RC, the less likely they are to have bad code.
When people wonder why problems like the recent jQuery flub manage to make it all the way into the wild, they tend to assume the issue is not testing enough. They’re incorrect. The issue is not testing enough.
Put the lead pipe down.
What do you think ‘enough’ is? I promise you, the majority of people who have a broken site will say “It needed more time being tested.” And in return, the majority of developers will say “It needed more people with more diverse setups testing.”
So which one is right? They both are. We need more of the right people testing. We need people who are on the edges of ‘normal’ for WordPress. The problem is that the ‘right’ people are the people who have the worst code.
Take a look at WordPress 4.5. Not a single beta tester was using a plugin or a theme that used jQuery improperly. Look at Jetpack 4.0. Not a single beta tester had a server config that broke, but within 12 hours they had over 300 tickets about an error 500 on many sites. In both cases, a significant amount of testing was performed. People banged on the code and outright tried to break it. Just not the right people. And the worst thing is we cannot know who is the right person until after something breaks.
This leads to the other part of the paradox.
Someone whose site breaks after an upgrade is less likely to want to beta test.
This is because we’ve broken the trust. They cannot accept that their site went down, or their computer crashed. Or maybe they can’t afford to be Janie on the Spot and jump every time their site needs a bit of an extra hand when an upgrade goes bibbeldy.
Of course, this is why the people who are testing are less likely to have a problem. They don’t want to waste time debugging stupid code problems, so they put the time in front to make sure they don’t introduce bad code to their environment.
We create a vicious circle, because the only way to get better results from beta is to get more people beta testing. But few people are willing to fully beta test on a live site, and say what you will, there’s nothing quite like live testing on a real, production site. You can’t stress test reality until you get your setup on your server on your design on your everything.
