30 Responses to “The Solution To The Lack Of WordPress Beta Testing”
This page contains comments from the The Solution To The Lack Of WordPress Beta Testing article.
Click the icon
to subscribe!
Subscribe
to subscribe!
Stephen Cronin is Manager of Online Service Delivery at a Queensland Government department & has been a freelance WordPress developer/consultant since 2007
*Content on this site is my own and is not related to my employer
Hire me - I'm expensive, but I'm very good!
WordPress Plugins
- DualFeeds: Offer both full & summary feeds to your readers!
- IFrameWidgets: Stop slow javascript widgets breaking your page!
- FeedEntryHeader: Add copyright info to the TOP of your feed!
- LocalCurrency: Show values to readers in their own currency!
- KeywordLuv: Reward commentators with keyword rich links!
- Comment Warning: Detect potential spammers and warn them!
- Eclipse Crossword Integration: Publish crossword puzzles!
- Ajax_the_views: Make WP-PostViews work with caching plugins!
- Remove Widget Titles: Remove widget titles on the front end!
Want a Custom WordPress plugin? See my Services page.
Greasemonkey Scripts
- Google Numbered Search Results: Number Google SERPs
- Bing Numbered Search Results: Number Bing SERPs
- Disable Ctrl-W: Disable the CTRL-W key so you don't lose work
- markItUp! for Greasemonkey: Enable markItUp! for text areas
Visit my home page at Userscripts.org.
I think that this is quite a good idea. I personally like to use the bleeding edge code, and most of the time I don’t purposefully go hunting for bugs. If I come across one then I will submit it.
As to how to physically implement this I’m not quite sure.
Would it result in a significant increase in traffic to Automattic servers since there is data being sent between the users server and the Automattic servers?
Hi Andrew,
Thanks for your comment.
That traffic is already be there. That’s how the each WordPress installation knows whether there’s a new version (that’s how you get the message to upgrade). It would be simply changing this message slightly, as well as the logic in WordPress itself, so it knows how to deal with the message.
This is certainly true.
Actually, now that I think about it, there might actually be a decrease in traffic since not everyone will be scrambling to update to the latest version.
Hey Stephen, your staged approach ala Google Labs, etc is a pretty cool idea. for starters, I would like for it to be opt-in. The opt-in could trigger a flag that lets the WordPress.org API know that I want to be part of the WordPress Live testing. Would this put me on Trunk or would this be a different version? I imagine it would have to be a little more refined than Trunk.
Interested to hear Peter Westwoods thoughts on this.
Hi Jeffro,
There’d be different ways to do this and I’m sure the Core team would have a much better idea about how to do it (I’m just throwing the idea out there) – but I wasn’t thinking of a new version, just holding back the “upgrade now” option for most people, while giving it to some who would then be testing the final release of the new version.
If it was opt in, then when beta testing was over, you’d get the upgrade message same as now, but most people wouldn’t get it for another week. That buffer lets any critical errors be found and a decision made about what to do.
And as I said, this would be bypassed for major security releases…
Interesting idea. I’d like to see that discussed with the core people
Hi Ozh – I’d like to see that too.
Try to insert the topic in an irc meetup / in the wp-hackers mailing list? Or maybe ask privately westi/mark their thoughts about this, to see if the idea has potential or is dead-born
Hi Ozh,
I’ve sent Westi and email asking him what he thinks… Thanks for the advice.
Brilliant idea ! Hope the WordPress dev team will implement this practice for futur WP releases.
Hi Nico,
Thanks and I’m glad you agree!
I think this is a brilliant idea, it would give WordPress a huge boost in the eyes of the enterprise. And just generally boost confidence.
Hi Oscar,
Great point about giving WordPress a boost in the eyes of the enterprise. I hadn’t considered that, but there’d be a much higher adoption of WordPress by the enterprise if they had more confidence in the testing process.
This is an excellent piece of informatioin and as a user (as opposed to a dev) I would like to see WP go this direction. For non-security crisis releases, I’d be willing to wait an extra week or two to get a better, less buggy, version.
The piece is also interesting because it is “long” and blog readers allegedly have an aversion to gaining full information from a long post. Your post spent the time/length to say what needed saying and I am glad you did not feel compelled to make it shorter. Not all useful information can be delivered as short jingle, real information often takes time to devlop and I am glad you did.
Your mention of Robert Scroble’s rant caught my eye. After reading it and other of his pieces I remain mystified why people pay him much attention. As I remain relatively new to blogging I wonder, did he used to be important or meaningful?
I found your site through WPTavern. Thank You and Semper Pax, Dr. Z
Hi John,
Thanks for your comment. Good to see that people would be willing to wait. Also thanks for reading through the whole post (and appreciating it). That’s just the way I roll – I break the topic down into sections, then expand on those.
As for Robert Scoble, he’s mainly famous because he started blogging when he was still at Microsoft. He became the voice on the inside of Microsoft, rather like Matt Cutts has done now with Google. The tech world looks to people like these who have the inside knowledge.
I think a staged release would have to be opt-in. Those who aren’t already opting in via the Beta Tester plugin wouldn’t like us opting them in without asking. Regardless, those who currently wait probably wouldn’t upgrade during the initial staged roll-out period even if they got the upgrade notice earlier than others. Without being able to forcibly upgrade people, we won’t be expanding the testing audience. The audience will still self select. Forced upgrades will never happen, of course.
The Beta Tester plugin already does pretty much what is suggested, and we encourage use of it in every beta and RC release announcement. Perhaps building it into WP would help, but there is a line to tread when building in nags.
Rolling out to a single site like gmail or wordpress.com is different than rolling out software that anyone can install anywhere. We often do staged rollouts on wordpress.com. That’s harder to do with self-hosted WP because we don’t control the process start to finish. We can force upgrades on wordpress.com and immediately fix any problems before rolling it out further. Neither is possible with self-installed WP.
I’m not sure that staged releases really help here.
They do help when you have full control over the infrastructure and you can release updates incrementally across your users.
To a certain degree the WordPress Beta Tester plugin allows you to already opt in to taking part in the process early and getting involved.
Staged releases as you describe them also go against the whole test before production philosophy which you, quite rightly, promote as a good practice.
In my experience working in a traditional software development team you can never get as many beta testers as you would like however hard you try to recruit them as they often do not understand the benefit they get by being involved.
I think the most important things we can do is make the availability of beta and rc as public and obvious as possible and make it easy for people to get onto the beta track and try to ensure that they get the recognition they deserve for their contribution.
Isn’t a staged release by definition released “incrementally”? It seems that allowing certain users as the author describes (100,000 for example) would just be ideal for the users that want to get the latest and greatest right away, while allowing more conservative ones to wait a little bit longer to reassure them.
It seems that these kind of discussions are always bias towards developers, it seems to be forgotten that most users don’t want to be part of “testing” because they usually feel like they have to do something special, or they simply don’t know what to do or what this means. Most people aren’t developers. Most people don’t want to “participate and contribute.” As selfish as this may be, it is the reality.
We already have a large group of testers, a large group of developers and hopefully large enough of a community to contribute early on and find bugs. Why not extend the tester group to a less sophisticated group that just uses WordPress, they may even dabble on their themes, but aren’t full blown developers.
I think the idea is great, let us get the latest available and let us acknowledge that it could have a bug, it could cause a problem. But in return you get that many more people testing.
People aren’t sitting around at home waiting to be “part of the beta” testers for WordPress, they’re waiting for the new feature that was promised so they can do something new or extend their website. They have their stuff to do, and it isn’t testing WordPress. So after we’ve done our early testing, and the release is ready. Why not let them have it as an RC and extend the user base just as the article suggests. Then you can make it an official release and make it available to everybody.
A simple way to make this “opt in” is simple. Something like “click here to receive the latest updates (beta, RC, etc)” with an agreement that this could break plugins/themes/etc. I think this is certainly better than releasing an update considering it “good” and finding that a bug went undiscovered, just to patch it with a .1 release a day/week later.
This would instantly extend the “beta group” and all these people have to do is use it. No special work required, no special downloads, no discussions in wp-hackers, etc. Most people using WordPress aren’t developers and don’t care to be. Make it easy to report crashes, even autodetecting issues and highlighting them on the dashboard, where a user could click “we found a problem, click here to send a report to the WordPress Developers” (maybe that’s a little overreaching, but hopefully illustrates my point). Then when satisfied, a release could be moved from RC to production.
Think about MS, how many people did they have in their beta testing for vista? 100,000? more? less? and it still flopped, why? They have the money, they have the resources… No real user input.
Gmail releases staged and also wordpress.com, and yes, they have control over their infrastructure and can easily monitor stuff, but this isn’t the case for Self hosted wp. I see this as one of the great advantages and a huge opportunity to make WP even better than it is. It means this for the development cycle, more people, more server configurations, different scenarios and stuff even you guys as developers haven’t even thought about.
There’s a gap between the official beta testers and the regular users, which can be filled with power-users or people that don’t mind trying out the latest, but don’t want to go through the trouble of setting up svn, going on trac, discussing their findings etc.
I don’t see how this would go “against the whole test before production philosophy” you’re simply extending the test to a huge user base and letting it run in the real world before you give it an absolute green flag. This is where I would like to go into my dashboard and say “no, only show me the latest production release” or “yes give me the bleeding edge, I’m willing to try it out” simply with the click of a button.
@Stephen, it was an uphill battle to have my company adopt WP, because it wasn’t considered “production” or enterprise level software. I think a formal process of development (as it exists now) is great, but would benefit from extended testing from the regular users. For example, we have our production site, and 3 or 4 clones of it as dev1, dev2, dev3, etc. All we do on those is test new releases, and I know I’m not the only one that does this, so if we had access to the new releases earlier we’d be a lot more confident.
eh… I think I went on long enough.
And some people tell me I’m crazy for waiting a long time for new WP installs.
My problem always is , many of my sites rely on hacked php files to work (Have you ever tried to kill the greeting messages on wordpress? It’s awful!)
It would be nice if they would just slow down the development and put out one GOOD version ,rather than 5 versions that sort of kind of work but make my plugins stop.
Interesting idea, something that the develppers of WP should really consider. I have never been a big fan of upgrading as soon as a new release is out.
Thanks for the in-depth discussion of software testing. I have always upgraded as soon as a new version is released (because it annoyingly begs me to at the top of the screen.) I haven’t had any trouble thus far, but will consider waiting after reading your article.
Great article.
I am fully agree with the post, that we must not go for the upgrade immediately after a new release of any software. There can any bugs or different sorts of problems with a new release. So, one must wait for few days before upgrading for any new release
Stephen-
As a long time developer myself (25 years, pc and mainframe) my only really true complaint of open source is the double edge sword of autonomy. WordPress is one example. WP is managed by a few, and the few have agreed to follow a particular set of standards. Some standards are generic enough that almost any developer with structured background can follow (naming conventions, commenting, using common functions, etc), but because of the nature of the idea of autonomy, one of WP’s advantages is that you can create a plugin to give WP functionality it currently doesn’t have (i.e. a plugin).
And because of the nature of open source, almost anyone with limited programming skills can contribute a plugin to the community. This is for the most part good, as you can start with a base, then customize it as you see fit. But from a corporate structure, this is bad, because each of the plugins are different, may or may not have been heavily tested, they may use naming convention, but not the Same convention. In addition to that complexity, not all plugins are maintained thru WP versions, so if you have one that works on, say WP2.71, but dies at WP2.92, and the plugin developer found a paying job in a different state, you are sol.
Not that it is good practice to use a plugin that isn’t getting regularly updated, but sometimes it is necessary. As an example, I was using a webhost who had a particular policy to not allow phpmail. I found an smtp plugin that worked. But as WP evolved, I had to modify the smtp plugin for each version because the developer had abandoned the plugin. I could have used a different supported plugin, but none of them actually did what I specifically needed. I eventually switched webhosts so I no longer needed the plugin, which took time to migrate code and data. (As a side note, had the plugin I used been incorporated in WP, it would have resolved the majority of email/hosting issues most users had).
In a corporate environment, that is a big ouch double no-no. In effect, you become a hostage of your own need to use unsupported plugins.
But more to your point on the testing. I think that any open source movement, such as WP could actually benefit with a little more corporate input and a little bit less autonomy. Best of both worlds, so to speak.
WP is great for non-techie indivuals to get out on the internet and toot their horn. It’s convenient, is loved by search engines, has a fairly decent admin interface, is customizable, requires very little technical background, and has a lot of community to support it (free themes, plugins, etc). But for the mid range extreme blogger/power users and small to medium business that have limited development resources, WP has a few major drawbacks, one of which is administration efficiency. Backing up the database – you either do an xml file, and it misses important things like links, or you do a full backup dump, but then if you have to restore it, it restores everything, including the junk left by disabled plugins that didn’t clean up after themselves.
Another example is – if you have, say, 500+ posts in 40 odd categories (which is very common), and you need to edit one or several in the middle, you can search for it if you know the name, or you can filter it by category – that is if you remember the name or category. Once you get to the post, do your edits and save it, you can’t go back to the filtered list. WP takes you back to the full list, where you have to start again. Besides the fact that WP has a poor sorting and filtering ability in just trying to track and edit posts, it is not very helpful as a high powered CMS that people are trying to us it for.
(One more rant)… The other thing that I have a hard time with is how pages and posts work, and how they sit in the database. In relational database theory this would be ideal, but in a physical relationship this doesn’t work. In essence, a post and page can exist in the same table and have different relationships to different sets of data, as long as the majority of the data of posts and pages is similar. But if a post and page calls different functions to produce different results, and if by changing the code on one side of the relation (changing a UI function to to address something, like tags), technically, the page side of the code needs testing, just to make sure something else didn’t break. Also, since the both sets of the data are in the same table, it should be easy to utilize either set of functions with the opposite code by supplying a simple data parameter (page/tags, posts/templates), but in WP, that is not the case without having to modify code and data for the most part.
I better get to my point. I believe that if WP and some businesses met to whiteboard some functionality/features that maintained backwards compatibility and added more business centric features (as in examples above); then WP would get more assistance in the testing department, and WP would grow in the direction that people are growing – and become a more efficient open source content manager.
Well, I’ve dawdled enough out here, time to git back to work.
Hi ScottydogDave,
I think that might be the best comment ever left on this blog. I agree with pretty much everything you said. I hadn’t even thought of quite a few of them.
I guess the other thing to add is that I think WordPress could do with one or two more experienced heads – nothing against the people running it now, I have a *lot* of respect for them, but I think it would be good if they were getting input from people who’ve been doing it for a long time, such as yourself…
Adding an option to install beta versions to the updater – what a FUCKING STUPID idea. This is what people running live sites use to update their installations – many of whom don’t really know much about what beta even means. People who are going to test beta version will do so with a fresh installation to avoid any risk of fucking their live sites up.
I honestly can’t believe how you can claim to be a Web Developer with such a truly retarded view of the processes involved in testing.
Hi Craig,
At least you’re just trolling and not spamming! I can handle that
Of course I said the solution’s not in the beta testing phase – that would still happen as it does now, but there’d be a staged rollout to everyone else rather than open slather.
It is always interesting to follow the WP discussion, and I agree that some stuff about the updated versions should be better communicated. However, WordPress is a strong product, and I am actually very fond of it. Thanks for posting!
And some people tell me I’m crazy for waiting a long time for new WP installs.
My problem always is , many of my sites rely on hacked php files to work (Have you ever tried to kill the greeting messages on wordpress? It’s awful!)
It would be nice if they would just slow down the development and put out one GOOD version ,rather than 5 versions that sort of kind of work but make my plugins stop.
Info Gadget – I agree with that for the most part – The Hello Dolly and Howdy thingy are totally irrelevant to any professional cms, and some of the hacks that are needed from the template side just don’t make sense. For example, the latest addition of custom menus is klunky at best, when a simpler hack is to just code page excludes in a template with a simple one line of code, storing the page ids in the option table (that way you don’t have to look up the pageids in each template file every time you change the menu).
I recently wanted to add the page/post thumbnail feature, but in order to get the feature to work on the edit page, you have to add an add_action in the functions template to trigger the thumbnails. Not only is the documentation weak about this feature, but unless you are a template developer, this one is easily missed. To get around this before it was added to 2.9, I just added a few lines of code in the functions.php, so when ever I want to add the thumbnails, they were there.
I also coded an admin tool to mass import thumbs for all posts. Now that 3.0+ has the feature, one can add the thumbs, but only if you add them for each page. (after you enable the feature in the template. Again, if you have 400+ posts, the last thing you want to do is manually add 400+ thumbs. There should be an easier way to import the thumbs for all posts in one easy swoop (I may share my code as a plugin one day).
If you pre-publish posts or use any kind of cron job, there is no way to view the cron jobs unles you use another plugin.
I have created some custom interfaces that don’t follow the regular blog look and feel. In doing so, I have ended up adding major php code to manipulate the content around the loop. I have also modified dozens of templates that had excess code dealing with the comments. If you turn off comments on pages from the begining, you shouldn’t need to tell the world that the comments are closed. The function should just break out of the comment loop. Also, there should be some defaults in the general settings to set global page/post comment and pings to off, and only turn them on as needed. To get around this, I created a hack (I guess you could call it a plugin), but now it seems I have about 20 plugins (hacks) that I add to each site just to get basic functionality. Add another ten or so for other features, and that is a lot of plugins that could fail during an upgrade.
With all the sites I have developed, I would guestimate that I have never used more than 30% of the functions that exist in WP for any website, and most of those functions I hardly use all the parameters/options that are available. It seems almost overkill.
On the other side of the coin, I do appreciate the effort the developers put in WP, as it has streamlined my development projects. I have cut development down from weeks to just days.
Later.
I’ve stopped upgrading completely with WP, and I have about 10 sites with them.
They’re a nightmare for releasing upgrades without being tested. The last one knocked out one of my sites completely, and I had to reinstall it on a different server and re-do half the information. Took me days.
I also now have 4 sites where the RSS feeds won’t work. Worked fine until I did the stupid upgrade. Now? No RSS feed and no way of figuring out why.
Now I advised everyone I know NOT to upgrade. WordPress is good for some things but these ‘fixes’ they do are often worse than useless.