Rockbox mail archiveSubject: Pre-release testing framework
Pre-release testing framework
From: Bertrik Sikken <bertrik_at_sikken.nl>
Date: Tue, 29 May 2012 18:58:53 +0200
Last week at devcon euro 2012 we talked about pre-release testing and
I'm volunteering to help things forward w.r.t. testing.
One of the problems we've seen with the last release was that really
basic functionality like audio file playback and radio playback did
not work on some targets and we were not aware of it until many days
later. Back in 2010 Björn already proposed a framework to recruit
the help of users to test release candidates in a systematic way:
This helps to at least be *aware* of any problems (what to do with
that information is another discussion).
I probably can't help with setting up a web-based framework, but
I can help with thinking up test cases.
The test case format I'm familiar with, defines the following
* some unique test case id, like
* a descriptive title, like
"Verify basic audio playback for all supported audio formats"
* (a reference to a requirement or spec, not really relevant for
rockbox I think)
* a pre-condition, like:
"The audio format test file set 1 has been copied to the player"
* action to perform the test, like:
"Using the file browser, browse to the test file set and select the
first audio file. Verify that all files play correctly."
* expected result, like:
"All files play and sound correct.
Failure criteria: * Audio playback shows audio artifacts (like
crackling, pitch or volume errors). * Audio playback is skipped."
(perhaps this is already an advanced test)
Stuff the tester can fill in:
* test result: PASS, FAIL, SKIPPED
* test remarks: free text
With respect to test strategy: In my opinion, we can start with
a set of rather basic tests to verify high-level behaviour, rather
than to go for test completeness. A test suite taking about (say)
one hour should keep the barrier low for people who want to
participate in testing.
What do you think?
Received on 2012-05-29