Testdays are a great way to get people to the playground; Bugdays inspire them to get dirty in the sandbox.

Recently, it was brought to my attention that the value of Bugdays were being called into question. For those who do not know, Bugdays are events used by QA to solicit community assistance in driving down bug queues and identifying potentially serious issues. I've given the idea of scrapping Bugdays a lot of thought over the last week, and the idea scares me. The only benefit to scrapping Bugdays, from Mozilla QA's perspective (as I see it) is that it would free up resources. This is pretty selfish in my view.

The Benefits of Bugdays

The benefits of Bugdays are numerous, and not just for Mozilla. There are benefits to the community as well.

Mozilla

The first benefit to Mozilla is that it helps us get our numerous bug queues under control. We are all owners of certain areas of code in Firefox. In fact, it is not uncommon for us to be owners of 2, 3 or more areas. Pile on release testing, update testing, crash analysis, organizing community events, writing automated tests, creating and executing test plans, and writing documentation. All this work quickly fills up our plates and leaves little room in a week for looking at bugs. Naturally, this results in ever-expanding bug queues. Bug days help us focus our efforts for a day and put a big dent in them.

The second benefit to Mozilla is that it gives visibility to bugs which have "no eyes" (ie. they are not on the radar of developers). There are thousands of bugs in bugzilla which are sitting in an UNCONFIRMED state. Now some of these bugs are on developers radars, but the majority aren't, or are on the periphery at best. Bug days gives a lot of eyes on these bugs for a day. Not one to forget history, there have been instances where blocker bugs have resulted from Bugdays. There have also been instances, albeit not often, that a blocker was discovered post release only to find we already had an UNCONFIRMED bug on file. Bugdays can prevent these drastic situations from occurring.

The fact that we currently have over 200 UNCONFIRMED and RESOLVED FIXED Plugin bugs to verify when we are making a push for out of process plugins scares me.

Community

The first benefit to the community is that it gives testers a higher sense of contribution. Certainly, this can't be said for all testers, but some seek something more technical and challenging. Most Testdays simply involve running a predetermined set of tests. This results in a feeling of assumed contribution (ie. testers don't necessarily see a the affect of their work on Firefox). They assume their testing efforts help make Firefox better. While this assumption is correct, it is less tangible than having a role in a bug's life. By nature, bug triaging is far more investigative than simply running tests. This appeals to certain people.

The second benefit to the community is that it reduces participation churn. As I said before, most Testdays follow the theme "come and run these tests". This is an excellent entry point to QA. However, for our more technically inclined contributors, this gets boring very quickly. In most cases, if we don't give contributors something else and something more challenging, participation begins to wane. I wouldn't be at all surprised if QA has lost contributors to Firefox development, Webdev, AMO, or non-Mozilla projects because QA hasn't challenged them enough.

The Fallacy

So the argument against Bugdays basically boils down to a cost-benefit analysis. In other words, does the cost (ie. usage of QA resources) outweigh the benefit (increased quality of Firefox). The root of this argument is in numbers. To be more specific, it is the number of contributors when compared to Testday levels. While numbers certainly do not lie (Bugdays usually have half the attendance of Testdays), the argument is flawed -- and this is why I think it's flawed.

Promotion

Firstly, Testdays are more heavily promoted. Now, some will argue with me on this point saying Bugdays are promoted more than in the past. This is certainly the case, however, Testday promotion is still higher. Testdays get an event page on QMO a month before then promotion a week, the day before, and the day of via newsgroups, forums, mailing lists, blog posts, social networking event pages, and twitter. Bugdays get a QMO event page two weeks before then promotion the day before on newsgroups and mailing lists.

Barrier to Entry

As stated earlier, triaging bugs typically takes a slightly more technical skill-set. For the most part, this skill-set can be learned, but it requires commitment from the contributor. On the other hand, Testdays require little or no knowledge of testing, programming or even Firefox. It's not hard to see that the pool of testers would be much larger for any given Testday when compared to that of a Bugday.

Release Collision

This is a big one. All community QA events require participation by folks at Mozilla. Internal participation is required for successful external participation. Unfortunately, the odds are stacked against Bugdays; Testdays are held on Fridays and Bugdays are held on Tuesdays. As many at Mozilla are aware, we have a pretty hectic release schedule. Most releases are scheduled to be done early to mid-week. It is "rare" (I use that term lightly) for a release to slip to a Friday. In many cases, if it does fall to a Friday, there will be so little work left over that it does not distract too much. If there is considerable work remaining on a Friday, the release schedule will often be adjusted to the following Monday. With Bugdays falling within the meat of this schedule, they often get ignored by people in Mozilla QA working hard on the release. Testdays are usually not affected much by this schedule as they fall on the Friday.

QA Activities

When Bugdays started many many years ago, it was one of the only ways to become involved with Firefox (apart from writing code). In fact, Bugdays (and bug triage) were one of the primary activities during my first internship at Mozilla and my initial involvement with QA. The result was a very high level of participation from the Mozilla QA community. Nowadays, there are so many different ways to become involved with Mozilla QA. We have Bugdays, Testdays, assisting with automation efforts, conducting exploratory testing, and helping with documentation, just to name a few. Just as we at Mozilla are spread thin across many projects, our community tends to get spread across many testing efforts. Since many in the community have day jobs (or school), the amount of time they can commit is far less than we can dedicate. This results in contributors focusing on one or two projects. Not that there is anything wrong with that, but Bugdays tend to fall short on this list, resulting in lower numbers.

The Case Study

So here's the "proof". Let's take the two most recent Bugdays. The most recent Bugday (Firefox 3.6.x Triage) resulted in a mere 11 bugs getting resolved. Compare this to the Seamonkey Bugday which resulted in 135 bugs being moved out of the GENERAL component into their appropriate components. So, the key Mozilla project (Firefox), and one that is due for a new release, had far less participation than a project which is completely community driven. How or why did this happen?

It is my contention that the 3.6.x Bugday had such low attendance due to the beta release work being done for Firefox 3.5.10 and 3.6.4. Admittedly, this kept me 100% occupied; I fully intended to attend the Bugday but release work kept me distracted all day.

What made the Seamonkey Bugday so successful were a few different factors. First, Kairo, one of the key project members, became deeply involved in executing the Bugday. He helped organize the community, got them involved, helped out by answering questions and assisting with the Bugday as a whole. Another factor was that there was not a lot of release work going on this week. This allowed for more participation from Mozilla QA.

Conclusions

Ok, it's been a lot of soap-boxing to get here, but we are here nonetheless. I think there is a lot of value in continuing Bugdays. I realize that we (Mozilla QA) have limited resources and that it is important to utilize those resources as efficiently as possible. However, I think it would be a severe misstep to drop Bugdays completely. Bugdays give our community something more challenging and tangible to participate in and it helps us isolate potential blockers (paramount in my opinion).

The root of all of this was a question that was asked, "Should we be dedicating QA resources to Bugdays?". In my opinion, this is the wrong question to be asking. We should be asking, "What can we do to increase community participation in Bugdays?".

The following is a list of things I believe we can be doing to improve participation and improve the value of Bugdays. In hindsight, most of these can/should be applied to all QA practices going forward:

     * Historically, we have thought of the "community" as those external to Mozilla who help us with QA.
       Our definition of "community" needs to be extended to those outside of our team.  Whether that be 
       other teams within QA or Firefox/Platform developers.  Actually, when I comes to Bugdays, especially 
       Firefox/Platform developers.

     * We need to get "feature experts", those people who have a deep knowledge of the product to become 
        involved at all levels of QA events.  This means project leads, developers, and community leaders.  
        Speaking to Bugdays, I think it is extremely important to have the Firefox developers on hand.  Not 
        only do they have expert knowledge of the feature, code and existing bugs, but they are the ones who 
        will be fixing any bugs identified by triagers.  Their knowledge and skills can be a great asset when 
        trying to reproduce bugs, especially when steps to reproduce are not clearly indicated in a bug or if a 
        bug requires a high level of technical knowledge.  Having a resident "expert" also helps to drive the 
        Bugday, is proactive in soliciting questions and helping testers greatly improves the results.

     * We should look at migrating Bugdays from an event we broadcast every month to a QA tool to be used 
        by different projects.  The idea here is to have different project leads come to QA and say, "Hey, we 
        have this backlog of bugs.  Can you guys have a look at them?".  This idea was brought to my attention 
        by Tracy Walker, project lead for Bugdays.  Naturally, I have some skepticism of the pro-activeness of  
        community and project leaders when it comes to QA.  This mostly comes from experience.  I think this 
        is mostly QA's fault though.  We have failed to showcase the value of Bugdays.  

     * We need to showcase the value of Bugdays and community testing efforts.  We don't do a good job of 
        communicating the benefit, value, and impact Bugdays have on Firefox.  In fact, I had to suggest a 
        Bugday idea to a veteran member of the QA team recently.  If our own team either ignores or has 
        forgotten the value of Bugdays, how can we expect any more from developers, project leaders, and 
        community testers?  A big part of this is a failure on QA's part to communicate the benefit of helping.  
       We do a great job of communicate what we need help with, but not how that helps and certainly not  
       why the help is so valuable.

Final Thoughts (aka the REAL Conclusion)

Going back to the cost-benefit analysis. It is quite possible that we currently put more cost in than the benefit we get out of Bugdays. I think it's folly to scrap Bugdays based on this analysis. I believe it signals that we need to get the ratio swinging more in the favour of benefit. Ultimately, to achieve this goal, we need to do a better job of communicating the value, usefulness and benefit of Bugdays; we need to get more internal involvement and feature experts in attendance; and we need to identify some high-profile Bugday topics to show developers, project leads, and the community that this is a tool, not just another event to be swept under the rug.

At this point, I invite you to take part in the discussion. This has all just been my opinion. However, I hope this has inspired you to start thinking about Bugdays as a tool and to formulate your own opinions on the matter. Should you wish to participate in this discussion, please comment on the mozilla.dev.quality newsgroup thread.

See you there, and thanks for "listening".