From: Frank Gevaerts <frank_at_gevaerts.be>
Date: Thu, 19 Nov 2009 20:22:25 +0100

Hi,

We've now done five releases in the modern era. I think that in general
the release process has worked well, but I do feel that there are some
problems.

A number of people spent large amounts of time to get 3.0 as good as
possible. This included work on the manual, basic functionality testing,
going through the bug list, trying to fix obvious bugs and possibly
disabling features if they were too buggy. There were of course
problems, and maybe the release process took too long (five weeks from
freeze to release), but I think that in general this was a nice release
that we can be proud of.

In my opinion, our latest release, 3.4, was very different. Some of the
people who had been involved in the release process were busy with other
things, and we kind of forgot about it. For a while it wasn't too clear
if the freeze had already started, and we didn't do any RC builds. The
result was a very short release process (one week). Quality-wise
however, I think (although I don't really have proof, this is more of a
general feeling) that this release was not very good, and that this is
just the low point in a longer trend, i.e. releases are gradually
getting worse.

So what's been going on?

I think there are two main factors involved:
- We've been getting used to doing releases, and working on them is just
  not as exciting as it once was
- More people use the releases, so bugs in relatively rarely used parts
  of the code don't get noticed immediately anymore

Can we reverse the trend?

I suspect that we can. The main thing that has been missing recently is
testing of the soon-to-be release. This testing does not need to be very
in-depth, often a quick check is enough to spot problems. It has to be
done on every target though.

My proposal is to involve our users here, and to do the following:
- Starting from the day of the freeze, we provide RC builds.
- We provide a feedback template that helps people figure out what needs
  testing, and to ensure that the resulting reports are reasonably easy
  to handle.
- Someone collects these reports and consolidates them to flyspray
  tasks for the actual bugs.

If we do this, having a week (or more, although I don't think more is
needed) of RC time after the freeze makes sense again. If we have two
weeks between the start of the freeze (and the start of the reporting
season) and the release, I suspect that we can fix most of the obvious
bugs in time, and if not actually know about them so we can mention them
in the release notes.

For the next release, I can volunteer for providing the RC builds, for
proposing the feedback template, and for collecting and analysing the
reports.

Thoughts?

Frank

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Received on 2009-11-19