Of Testcases and Learning

A couple of weeks ago we held the first ever week-long bugday-like event. Essentially, it was a week of workshops, writing test cases, bug triage, and testing new features. It was an effort to organize and execute, but it was very rewarding too. Thanks largely due to the efforts of those who donated their time to help and contribute to Firefox.

Community Feedback #

Before starting each day, we asked new community members to fill out a survey. One of the questions was designed to determine their motives for joining the event. Almost every member of our community responded, “I love Firefox and want to make it better” and , “I want to learn something about testing”.

Based on these responses, it came at no surprise to me that the participation was highest during times of focused training and new features. The community responded well to workshops and being able to work one-on-one with someone in MozillaQA. In addition, we held competing events on Friday; a bugday and a testday. The level of participation was fairly standard for both of these events, but people seemed to be more focused. There was concern that competing events would divide and confuse the community. On the contrary, it gave them choice. Going into the next quarter, I’d like to run multiple events. We could run events which focus on training; events which focus on testing a particular feature; events which focus on specific bugs and bug activities; and events which incorporate aspects of all three. I think running these types of events adopts an approach which is inclusive to the wide range of community members we see event to event.

Speaking to that wide range, it is interesting to note that 55% of the community members were from outside of the US and did not speak English as their first language.

From this, I think it is worthwhile to investigate localization of event notices and documentation. In addition, we need to investigate and experiment with events being “open” up to 24-hours. I don’t think it will be an easy thing to achieve and will be something we have to iterate and transission towards. It will require a team of leaders from many different regions. However, I think it is certainly achievable.

Advertising #

Due to the extreme workload that goes into planning a week-long event, everything was not in place until 12 hours before the event was due to start. As a result advertisement was given a very small window. I think we need to experiment with a more frequent and reliable event schedule, like an event every Friday, and we need to advertise more often.

Going into the next quarter, I’d like to experiment with an advertisement schedule similar to the following. On the first of every month, we should send out a summary of events coming up in that month, either on QMO, in a newsletter, or both. The week of the event (probably Monday) we send out the verbiage for the event on multiple channels; a blog post on QMO, a tweet, an event page on Facebook, posts to various mailing lists, and even change the topic in the IRC channel to advertise the event. The day before the event, we tweet, post a reminder on QMO and the mailing lists, and send out a broadcast on IRC (once at 9am, 1pm and 5pm PDT). The day of, we can tweet and broadcast on IRC at 2 hour intervals throughout the event. Finally, on the day after, we post the results on QMO with a mention of the next event.

One further action item which we are investigating this coming quarter is the effectiveness of our existing communication channels and whether there are some which remain untapped. I’ve touched on many of the existing channels we use above, but I’d be very interested to hear from you if there are any ideas you have for communicating with the community. Keep in mind, I’m broadening the definition of community to include both getting the word out and communicating in real-time with contributors.

Documentation #

In order to execute this event effectively, it took a lot of documentation and tracking tools. These tools ranged from wiki pages to Google Docs. Going forward, I’d like to migrate the tools which were useful into a template system which people can tap into to run their own events. Basically, I’d like to create a “portal” to aid people in planning, communicating, executing, tracking, and reporting their events. Creating these documents and tools will largely be an iterative process, implementing feedback from event to event.

In addition, I’d like to create a set of documents we can point community members to if they want to work outside the confines of an official event. For example, if someone new hops on IRC and wants to learn how to write a test case, we should have a document set up which allows them to learn how to write a test case.

Tools #

QA has several tools, some used more than others. These tools include the QABot in IRC and the QA Companion Firefox add-on. We will be investigating their effectiveness, how they can be improved, and whether there is a need for new tools. For instance, one of the failings of the bugweek activities was that many found the input method to be cumbersome. The general process was as follows:

1. Find a bug in bugzilla
2. Write the testcase in a spreadsheet
3. Ask for review on IRC
4. Get feedback
5. Make revisions
6. Repeat step 3-5 until the testcase is good
7. Mozilla person adds test to Litmus
8. Mozilla person updates spreadsheet
9. Mozilla person updates bug with comment, in-litmus flag, and whiteboard

Without speaking to the actual tools used in this process, and speaking to the process itself, it seems perfectly evident that a single tool could be used to do most of the legwork. This is something we will be investigating.

Summing it all Up #

Overall, the bugweek was great. I can’t thank the contributors and people who helped enough. The week wouldn’t have been a success without them. In addition to the new tests contributed to Litmus, we received a lot of great feedback. We came out of the Summit at Whistler with a lot of great ideas regarding community engagement. It took a lot of work to organize the bugweek but it gave us the opportunity to try out some of those ideas and learn some lessons. Going forward, we will be tweaking the formula created by the bugweek and trying out some more ideas.

Thanks everyone for your help. I hope to see you around IRC soon.

---

Results of the Bugweek #

94 bugs triaged

• Bugday: 61 bugs
• Testday: 33 bugs

45 testcases in Litmus

• New: 38
• Revised: 7

Contributors: 11