From: Bertrik Sikken <bertrik_at_sikken.nl>
Date: Tue, 29 May 2012 18:58:53 +0200

Hi all,

Last week at devcon euro 2012 we talked about pre-release testing and
I'm volunteering to help things forward w.r.t. testing.

One of the problems we've seen with the last release was that really
basic functionality like audio file playback and radio playback did
not work on some targets and we were not aware of it until many days
later. Back in 2010 Björn already proposed a framework to recruit
the help of users to test release candidates in a systematic way:
http://www.rockbox.org/mail/archive/rockbox-dev-archive-2010-10 /0114.shtml
This helps to at least be *aware* of any problems (what to do with
that information is another discussion).

I probably can't help with setting up a web-based framework, but
I can help with thinking up test cases.

The test case format I'm familiar with, defines the following
properties:
* some unique test case id, like
  "basics.playback.001"
* a descriptive title, like
  "Verify basic audio playback for all supported audio formats"
* (a reference to a requirement or spec, not really relevant for
  rockbox I think)
* a pre-condition, like:
  "The audio format test file set 1 has been copied to the player"
* action to perform the test, like:
  "Using the file browser, browse to the test file set and select the
  first audio file. Verify that all files play correctly."
* expected result, like:
  "All files play and sound correct.
  Failure criteria: * Audio playback shows audio artifacts (like
  crackling, pitch or volume errors). * Audio playback is skipped."
(perhaps this is already an advanced test)

Stuff the tester can fill in:
* test result: PASS, FAIL, SKIPPED
* test remarks: free text

With respect to test strategy: In my opinion, we can start with
a set of rather basic tests to verify high-level behaviour, rather
than to go for test completeness. A test suite taking about (say)
one hour should keep the barrier low for people who want to
participate in testing.

What do you think?

Kind regards,
Bertrik
Received on 2012-05-29