The Solution To The Lack Of WordPress Beta Testing

While catching up on some old podcasts, specifically Episode 82 of WordPress Weekly, I came across a discussion about WordPress beta testing. The discussion centers around the problem of bugs not being caught during beta testing because there just aren’t enough beta testers.

To me, the solution seems straightforward – but that may be because I worked in the software industry for 10 years and have experience in software release management, so I’ll take the long path and set the scene properly.

That makes it a long post, but stick with it – you need to read it all to fully understand my reasoning. First we need to consider the following question:

Should You Upgrade WordPress Immediately After A Release?

Short answer: Sometimes, but not always.

Over the last couple of years, there have been repeated calls for WordPress users to update their blogs immediately after a new version has been released. At times, the call to upgrade has almost become a frenzy.

I’ve taken this with a grain of salt and held to my belief that upgrading immediately isn’t always the correct choice. Sure, upgrade immediately if the release fixes a critical security issue – but most of the time it’s better to wait and see whether:

• there are any major bugs in the new version
• your theme works with the new version
• all of the plugins you’re using work with the new version

I’ve developed this ‘hold back’ approach while working in the software industry over the last 15 years or so – I’d contend that it’s ‘best practice’. Upgrading to a new version without testing the new version in your environment (including server, theme and plugins) is the sign of immature processes.

Some may argue that this approach is overkill for a simple blog, but there are many sites out there running WordPress that are a little more complicated.

Even simple blogs are often not that simple, especially as so many are now monetized. Ask yourself, will I lose money if WordPress doesn’t work after an upgrade? If the answer is yes, you should probably be testing new versions before upgrading.

You’re probably asking how does this relate to beta testing? Well, in a couple of ways:

1. Bugs In New Releases Mean Less People Will Upgrade

Let’s leave aside the fact that I don’t personally subscribe to the ‘upgrade immediately’ advice and assume that it’s a good thing (which it would be in the ideal world).

Bugs in new releases undermine this advice. After the recent problem with scheduled posts, introduced in the WordPress 2.9 release, there have been people questioning whether upgrading immediately was a good thing.

If beta testing was improved and caught more of these problems, then there’d be less of the ‘hold back’ people out there and more people would follow the advice, in theory making WordPress more secure. There’d be:

• less people (such as Robert Scoble) complaining about WordPress security.
• less need for Matt to write public responses outlining why people should upgrade.
• less complaints about bugs and the loss of functionality.

There’ll always be bugs in software and there’ll always be people who complain – but more comprehensive beta testing would lead to less bugs, leading to less unhappy users, leading to more sites being updated immediately, leading to less hacking, leading to even less unhappy users.

Better beta testing begets better security.

2. Balance Between Upgrade Speed Vs Security

Alternatively, if we don’t leave aside my beliefs, if we can moderate the unthinking update immediately advice, then we’re provided with a tremendous opportunity:

On a case by case basis, we can decide if an upgrade is in the “update immediately” category or in the “can wait for critical bugs to be fixed” category.

My solution, way down the bottom of this post, is going to propose delays to upgrades that are not for critical security issues, so that user experience less critical bugs. But more on that later.

The Problem: A Lack Of Beta Testers

The problem, which has been mentioned in various places including the podcast, is that although WordPress has a beta testing phase, there aren’t enough end users testing it to catch all the critical bugs. The software appears to runs fine during beta testing, but bugs rise to the surface when the release is rolled out to everyone.

First, some facts:

• Software is always going to have bugs
• Beta testing will never find all the bugs

There will always be some bugs that are only triggered when the software is run in an obscure environment or is used in a way that’s slightly different from how most people use it.

So it’s not about eliminating bugs, just about reducing the number of critical bugs that get through beta testing. The only way to do this is to get more people, using the software in different ways, on different server environments, to test the software.

How do we get more people to test the software? The answer’s coming soon (honest).

Software Release Management Anecdotes

First, let me say that I worked in the software industry (for a library management system vendor) for 10 years. For quite a bit of that time, I was the person who authorised the release of software to clients and/or agents.

I could bore you with software release management anecdotes going back to 1994. Luckily for you, I came to my senses and edited them out! I’ll just say that making sure that major releases were properly tested and free of bugs was extremely important for the following reasons:

• Loss of reputation, harming future sales: Upset users are significant in the relatively small library software marketplace.
• Direct financial loss: yes, it cost a significant amount of money to burn 5000 CDs back in 1994 (and then post them out).
• Lots of additional work: printing letters, packaging CDs and supporting users who were upgrading, etc.

Obviously, WordPress operates in a very different environment from library software industry of old and many of these points are less of an issue:

If WordPress lose one user, or even ten thousand users, because of loss of reputation, they don’t actually lose any money in sales (actually there is no they!). If they have to release an upgrade, there are no costs for CDs and postage, people just download it. There’s no surge in the amount of support they need to provide, as support is crowd sourced.

Does that mean WordPress can afford to ignore this issue?

I say No – they should care about their reputation (and their users), even if there isn’t the same financial emphasis that there is in the closed source software industry. In fact, they should look to the closed source software industry and learn from it. Solutions to this very problem have been worked on and refined for decades.

In my particular case, we had several major incidents that led to us refining our testing and release processes. We ended up moving to a process that involved three phases of testing – I’ll outline these and relate them to WordPress in the next section.

The Three Phases Of Testing

1. Developer Testing

I’m sure this is done comprehensively by the WordPress developers. Ultimately, we employed a couple of full time testers, which Automattic could possibly consider funding. They may already have some testers (besides the Chief BBQ Taste Tester), although it’s not obvious from their About page.

But the problem isn’t here and full time testers would only have limited impact, so let’s keep moving.

2. Beta Testing

This is being done, but the problem is that not enough end users are involved. WordPress could look at ways to entice more users to join the beta testing program, but many of the methods that the closed source software industry use, such as offering discounts to beta testers, won’t work in the open source world.

One idea could be to build in a "do you want to upgrade to the beta version (for the greater good)" message into the automatic update function. When a beta version is released, everyone gets this message.

It would have to be very clear that it was a beta version and that there may be some problems and it would need Yes, No and Never Ask Me Again options, but it could work. Not many would choose to upgrade, but 1% of 3 million is 30,000, which is a lot more than the number of current beta testers.

Once again, though, I don’t think the answer is here. It would be a significant improvement, but could cause a lot of negative publicity: there’d be too many people who’d choose it accidentally, then blame WordPress for upgrading them to beta software.

3. Live User Testing / Staged Release

So we come to the solution at last (I told you we would) and it’s not in the beta testing phase.

Instead, the answer lies in a staged roll-out to users: limiting new releases to a relatively small number of people until it’s proven that there are no major problems.

If any major problems are found, a decision can be made on whether they are serious enough to fix immediately, before making the software available to the rest of the users. Even if it’s decided not to fix the problems, users can be made aware of issues that may affect them before they upgrade. Even that small improvement would make a real difference to users.

The staged release approach been used for decades. I was using it 15 years ago. Google uses it today: they roll-out new features in Google Analytics or Google Adsense slowly.

How Would A Staged Release Work?

A staged roll-out of WordPress would work something like this:

1. Once the beta testing phase is complete, the automatic upgrade notice would be sent out to say 100,000 people.
2. There would then be a period of say one week, where the feedback would be monitored and any critical bugs identified and investigated.
3. At the end of the period:
   • if no critical bugs have been identified, then the automatic upgrade would appear to everyone else.
   • If critical issues have been identified, a decision would be made on whether to proceed with the release or delay it until the problem is fixed.

It goes without saying that urgent security releases would bypass this.

Now I’m not talking about physically denying the software to anyone, just manipulating who gets it through the automatic upgrade process. Anyone would still be able to visit wordpress.org and download the lastest version.

How Would A Staged Release Be Achieved Technically?

I’m not the best person to answer that, but I’d imagine it wouldn’t be too hard to do. It would require the automatic upgrade function to be modified: instead of WordPress installations asking "is there a new version", they’d ask "is there a new version and can I have it".

On the server side, it would check the download count and, when it reached the threshold, start saying no. You could make it complicated (checking whether it’s been downloaded from this IP address before), but it would be best to keep it simple.

There exists the potential that some users may be unhappy about either getting the new version before it’s been through live testing, or having to wait until after. The key here is that users are not getting the beta version, they’re getting the real version that’s been through beta testing, just as they would now. It’s just that it’s being rolled out slowly to further protect users.

This would need to be well communicated.

Of course, it could be made it opt-in. The first 100,000 users could be shown the message: "WordPress 3.1 is available! Update or Wait". If they choose Wait, it won’t ask them again until the live release testing is complete. The rest of the users would be shown: "WordPress 3.1 is available, but we recommend you wait for moment. Wait or Upgrade anyway".

Final Thoughts

So the solution to the beta testing problem isn’t actually within the actual beta testing phase. It’s just to roll out the upgrades slowly, giving time for critical bugs to be caught before they affect too many people. This would protect the majority of users from any serious bugs.

Personally, I have doubts that this would ever be adopted: I think it would be seen as too complicated. But really, it wouldn’t be that hard to do and would make life better for the vast majority.

What do you think? Am I crazy? Do you disagree with me? Do closed source practices have no place in the open source world? Do you have a better idea? Let me know!

Tags: security, software, testing, WordPress