Everyone knows what Fedora QA does. "They're the testers. They test stuff." Sure, that's true. But.. what, exactly, do we test? And how?
That's where Test Plans come in. A Test Plan is a document that tells the testers how to test something.
A good software Test Plan should at least answer these four questions:
- What special hardware / data / etc. is needed to test this (if any)?
- How do you prepare your system to test this? What packages
need to be installed, config files edited, etc.?
- What specific actions do you perform to check that the software is
working like it's supposed to?
- What are the expected results of those actions?
Your answers can be short: "get a bluetooth keyboard and yum install bluez-gnome newer than 0.25" answers question 1 and 2 just fine. But they need to be complete and explicit: "Get an appropriate keyboard and install the new packages" is not helpful.
Questions 3 and 4 are answered by Test Cases. A Test Case is a single, specific thing to test - a list of actions to perform, and the expected results of those actions. Depending on how complicated a package or feature is, you could have one Test Case or a whole bunch of them.
While writing a Test Plan, pretend that you have an intern whose job it is to test Fedora releases. This brave, brave soul has a few years' experience as a Fedora user and basic familiarity with using the commandline to configure stuff, install packages, and so on.
Your job is to tell this person how they can test your favorite package or your cool new feature, so they can either a) tell their friends about how great it is, or b) tell you (via bugzilla) if it breaks.
Here's an example Test Case for Banshee, originally written by our own Balaji Gurudass:
This gives us a clear, easy-to-follow way for anyone to prove that a new banshee package does (or doesn't) work like it's supposed to. Or, at least, one small part of it.
Test loading a Local File in Banshee.
- Start Banshee
- Choose "Import Media" from the "Media" menu
- Select "Local Files" from the import source dropdown and click the "Import Media Source" button
- Select the music file and click Open
- Find the song in the music library and play it.
- Banshee starts successfully
- The "Import Media" menu item brings up the "Import Media to Library" dialog
- The chosen music file is imported into the Banshee library
- The music file can be played and the tracker moves along and shows time elapsed.
So there's an overview of how to write test plans. Currently we're keeping Test Plans in the wiki. There's a section in each Feature Page for a Test Plan, and if you wanted to write a Test Plan for a specific package (or set of packages), you could add a wiki page named "QA/[package] Test Plan". For example, here's QA/Banshee Test Plan.
Next time, I'll talk about why I think this stuff is so important.