There’s a weird aspect to open source that can be hard to explain, and that is the ethos and practicality of Beta Testing. Beta Testing is one of the most important aspects of open source, because it’s only with the beta that we’re able to real-world test products. No matter how much planning and testing and automated tests you write, there is nothing quite as powerful as the real-world. We just can’t reproduce it well enough because, among other things, we can’t yet predict humans very well. Sure, cold reading is a thing because humans can be predictable, but that doesn’t mean we know how a tool will be used 100% of the time.
And being a beta tester is hard work. You have to be both a user and a developer, which are really two totally divergent mindsets. Testing as a user versus testing as a developer is as different as how you and a formula one racer drives a car. Input from both the users and the developers is critical to the growth of a project and the end results. But before we get into how one really betas, we have to ask a very important set of questions.
Should I be a Beta Tester?
- Are you brand new to the project?
- Do you use the products daily (or close enough)?
- Do you know how to manually clean up a bad install without losing your data?
- Do you make regular backups?
- Do you know the basic troubleshooting steps for the product like the back of your hand?
- Are you willing to test on your live site?
If you answered ‘no’ or ‘maybe’ to any of those questions, then you’re not yet ready to be an effective beta tester.
That last one may catch people by surprise. Testing on your live site is something we actually tell people not to do, but I’m here to tell you that the best beta testers are either testing on their live site or on a site they use every single day. Since I have multiple site I use daily, I picked the one I can live with if it dies and I use that for my live-beta.
Guess what? You’re on it right now. It’s a hallmark of how much trust I have in WordPress. I can count the issues I’ve had on one hand since I started this a few years ago. I make a backup every day, twice a day, and I have it in three locations. If my site goes down, I lose at most 12 hours of content, and for what I’m doing here, that’s okay. My other sites, I could not deal with that loss and thus I use stable. But I test with live because… well let me explain.
How do I test?
If you’re going to be a beta tester, then the you install the beta and you use the product.
You were expecting rocket science?
Most projects have a lot of automated tests and they’ll test what they know about and what they can. But absolutely nothing replicates real world use. The whole reason I test with a live site, and why I feel it’s important, is that this is as real as it gets. This site is updated regularly with plugins and theme changes. I add new content five times a week on the same Multisite installation. I add new content multiple times a day on another site. One is on trunk, one is not. This lets me constantly compare the experience between the two and makes me immediately aware that something is different. If it gets my attention, that’s important.
What I do with my site is equally important. I’m writing content, constantly, which means I’m using some of the fundamental, day to day features that WordPress absolutely must have. That kind of user testing cannot be scripted. There is no AI yet that can reproduce what machinations a human gets up to.
I like to tell a story about the time my coworker John and I were testing some tax software for the bank. We ran it through it’s paces, putting in data, and every time we tries to save at a certain point, the software crashed. It crashed hard. We called the vendor and talked to them about it, walking them through what we did, and lo, it crashed again. They couldn’t reproduce it, so we screen-recorded it and sent it over. They were astounded and sent out a tech from New York to Chicago to work on this. He couldn’t solve it, but he did see how we crashed it. Then they sent in their big guns, their lead developer, and he sat down and watched us. As soon as it crashed he said he knew what the problem was.
We were doing it wrong. We were entering bad data in a way no tax professional ever would, the system was trying to process the math and, because the logic was bad, it crashed. I pointed out that people would make mistakes and he agreed, saying it should have alerted us to an error in the form, not try to process anyway. A few days later, he sent us a new version which properly trapped the error.
While a scripted test might have caught that by looking for bad values before mathing, they were doing the obvious check. They checked if it was a number. They neglected to check if it was a permitted number. If you’ve ever heard me nag someone about proper data sanitization and validation, this is why. They made a code change, added it into their scripted tests, and learned and grew. But they couldn’t have done that without a real human thinking in a different, and unexpected way.
We can, and will, improve scripted tests, but they will never improve to perfection because humans are pretty crazy. Which leads us to the next step.
How to I report a bug?
This is generic for a reason.
First, make sure you can reproduce the bug. Get a clean build to test on and go through the steps again. If you’re not sure about the steps, make a note to yourself about generally what you remember and try it again. If you can’t reproduce it, call it a one-off and let it go (and debug for yourself, but that’s different). Once you can reproduce it on a clean build, document those steps.
You’re going to want to answer these questions:
- What were you doing?
- What did you expect to happen?
- What happened?
- What research did you do into the error?
- How do you reproduce it reliably?
Include any error messages. Explain what you did. Don’t say “It didn’t upload.” Instead, say “The upload hung, making no further changes on the page. I waited 15 minutes before hitting refresh. When I checked the media library, I saw the image was uploaded however when I looked at the file server, none of the thumbnails were there.”
This is why you need to know what you’re testing. With a failed image upload on WordPress, you should know that the image uploads and then it makes the resized images. Even if you don’t know that, you should know your images are in wp-content/uploads/2015/02/ and you should look there to see if you can find them. If you have a failed post, you should know to look at the post list page in WP Admin and see if the post is listed there.
You’ll get extra bonus points if you can find your PHP error logs and share pertinent information, but that isn’t always easy. When you reproduce the error, make sure you specify if anything special has to happen. Like if the image upload only fails if you’re uploading a PNG, note that. Or maybe it only fails on pages, but not posts. How weird is that, right? Note it.
Don’t worry about being technical here. Be accurate, be clear spoken, and assume the other person is relatively new at whatever you’ve broken. Don’t assume they know exactly what you mean when you said “Upload an image…” Be specific and say “Create a new post and press the ‘Upload’ button…” If you know there are multiple ways to upload an image, test those other ways. It’s a due diligence thing.
After you’ve reported the issue, keep informed. Make sure you get email alerts for it, make sure you reply to those emails for it. You can’t just report it and walk away, you have to keep tabs and pay attention. People may need you to clarify information and explain problems in different ways. Remember people can’t actually read your mind. What’s clear to you, because you did it, may confuse them.
And finally … be prepared to hear that it’s just not that big of a deal. Sometimes a bug isn’t a bug, but an intended change. You may not like it. Heck, you may despise it, but that doesn’t make it wrong.
What else?
What are your takeaways from beta testing? How do you do it?
