At UDS-Q we will be discussing the relatives merits and deficiencies
of the bug triage process that we have been using in the server team
for the last 12 months or so (see [0]) as well as decided on how to
approach the current backlog of bugs.
FYI people on the weekly triage rota normally use the triage report
[1] to identify bugs to work on for their allocated slots.
I'd be interested in hearing peoples opinions on the current process
in advance of UDS - specifically what you think works and what does
not and how you think things might be improved.
Regards
James
[0] https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#triager
[1] http://status.qa.ubuntu.com/reports/ubuntu-server/triage-report.html
- --
James Page
Ubuntu Core Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
On 03/05/12 16:23, James Page wrote:
> Hi All
>
> At UDS-Q we will be discussing the relatives merits and
> deficiencies of the bug triage process that we have been using in
> the server team for the last 12 months or so (see [0]) as well as
> decided on how to approach the current backlog of bugs.
>
> FYI people on the weekly triage rota normally use the triage
> report [1] to identify bugs to work on for their allocated slots.
>
> I'd be interested in hearing peoples opinions on the current
> process in advance of UDS - specifically what you think works and
> what does not and how you think things might be improved.
>
> Regards
>
> James
>
>
> [0] https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#triager [1]
> http://status.qa.ubuntu.com/reports/ubuntu-server/triage-report.html
- --
>
James Page
Ubuntu Core Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
05-07-2012, 06:41 PM
Robie Basak
UDS-Q Topic: Server Bug Triage Process
There were two issues I wanted to bring up. I apologise for not managing
to send this email before UDS - I was away last week. So this is a bit
of a brain dump.
1. Triage list does not reduce after triaging is complete.
Bugs that can be worked on out of a triage list will inevitably be
progressed. Bugs that cannot be worked on out of a triage list will stay
there. So any list which permits bugs that cannot be worked on will tend
to fill up with bugs that cannot be worked on. This can make such a list
useless. So bugs that cannot be progressed must somehow be prevented
from appearing in any triage list.
So with this requirement in mind, I think that the current issues are:
a) Many bugs currently stay on the triage list because we can't progress
them. In particular, this includes bugs that are reported adequately but
not confirmed and with no instructions to reproduce. This is a
reasonable bug state, but unfortunately triagers can do nothing with
them. I think that these bugs are what cause most of the triaging
difficulty at the moment. For the purposes of this discussion I'm going
to call these bugs "awkward" bugs.
b) Bug importance is very difficult to select for awkward bugs. This is
because the real importance depends on the impact of the bug, and at
this stage this is often unknown. Importance can of course be changed
later, but setting a bug to low importance currently makes it unlikely
that it will be looked at again soon. If a bug is reported once,
initially appears not to be severe, but consequently turns out to be
very important, it could be missed.
c) If we set the Importance to remove awkward bugs from our first triage
list, all we are really doing is shifting the problem to a future triage
list. Really the awkward bugs should not appear on a triage list at all
until they become non-awkward (eg. get confirmed, or somebody posts
further information or steps to reproduce).
What I'm looking for is a way to say "this bug looks fine but cannot be
worked on until something happens that is out of the control of a
triager". Currently we keep state only in bugs, which is perhaps a good
idea. But it seems that we're unwilling to mark this state as such in
the bug, perhaps in order not to discourage the community from
submitting them. What would be really nice would be some kind of tag or
marker that automatically disappears as soon as a bug is changed in any
way so that it gets the attention of a triager again.
So our options are: 1) maintain state outside of the bugs themselves, 2)
mark the "cannot be worked on state" inside the bug using a tag, or 3)
be forever plagued with bugs in our triage lists
2. Automatic submission of package maintainer script failures
2a. Inadequate bug description often makes these very difficult to
triage, and they stay on the list. Often the description is so vague
that it's difficult to even prompt for further information without
turning the bug into a full one-on-one support channel which we don't
really have the bandwidth to do. I propose that these bugs are managed
separately from regular triage, and/or have a standard "inadequate
report" type standard response, but written specifically for maintainer
script failures (since the usual one doesn't seem very appropriate for
these cases).
2b. The reporters of most of these bugs with inadequate descriptions
appear not to be interested in taking them further.
2c. Most are support requests and not bugs, reducing the time that
triagers get to triage actual bugs.
2d. Desktop package maintainer scripts are expected rarely to fail, and
if they do it probably is a bug. Server package maintainer scripts often
try to set up daemons, but users customise daemons beyond what
maintainer scripts can be expected to understand, causing them to fail
and encourage reporting as bugs. And desktop users often install server
packages which then confuses the matter when they try and upgrade.
Linked with 2c, this causes a big problem for bug triage.
Of course, there is no real distinction between server and desktop
packages expect in the nature of what they are expected to do, and thus
what their maintainer scripts try to do.
I think it would be appropriate to see these bugs dealt with
automatically (boilerplate response) unless or until:
1) It's a "proper" bug report where the submitter has actually spent
some effort in reporting it (ie. with a bug description on par with
what we'd expect for a manually submitted bug).
2) Aggregrate analysis suggests that the bug is real and thus we
should look into it.
3) We've caught up with all the other bugs.
At this point, I suggest always changing the subject of the bug from the
automatic subject line to one that describes the real problem. This
would really help to make the real bugs stand out from the automatic
ones.
The boilerplate would of course explain the situation with instructions
on how to get the bug looked at if the reporter wants to take it further
(eg. a variant of
https://wiki.ubuntu.com/Bugs/Responses#Not_described_well)
Robie
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
05-07-2012, 07:28 PM
Serge Hallyn
UDS-Q Topic: Server Bug Triage Process
Thanks, Robie, all well thought out and stated. I agree with each point
here.
-serge
Quoting Robie Basak (robie.basak@canonical.com):
> There were two issues I wanted to bring up. I apologise for not managing
> to send this email before UDS - I was away last week. So this is a bit
> of a brain dump.
>
> 1. Triage list does not reduce after triaging is complete.
>
> Bugs that can be worked on out of a triage list will inevitably be
> progressed. Bugs that cannot be worked on out of a triage list will stay
> there. So any list which permits bugs that cannot be worked on will tend
> to fill up with bugs that cannot be worked on. This can make such a list
> useless. So bugs that cannot be progressed must somehow be prevented
> from appearing in any triage list.
>
> So with this requirement in mind, I think that the current issues are:
>
> a) Many bugs currently stay on the triage list because we can't progress
> them. In particular, this includes bugs that are reported adequately but
> not confirmed and with no instructions to reproduce. This is a
> reasonable bug state, but unfortunately triagers can do nothing with
> them. I think that these bugs are what cause most of the triaging
> difficulty at the moment. For the purposes of this discussion I'm going
> to call these bugs "awkward" bugs.
>
> b) Bug importance is very difficult to select for awkward bugs. This is
> because the real importance depends on the impact of the bug, and at
> this stage this is often unknown. Importance can of course be changed
> later, but setting a bug to low importance currently makes it unlikely
> that it will be looked at again soon. If a bug is reported once,
> initially appears not to be severe, but consequently turns out to be
> very important, it could be missed.
>
> c) If we set the Importance to remove awkward bugs from our first triage
> list, all we are really doing is shifting the problem to a future triage
> list. Really the awkward bugs should not appear on a triage list at all
> until they become non-awkward (eg. get confirmed, or somebody posts
> further information or steps to reproduce).
>
> What I'm looking for is a way to say "this bug looks fine but cannot be
> worked on until something happens that is out of the control of a
> triager". Currently we keep state only in bugs, which is perhaps a good
> idea. But it seems that we're unwilling to mark this state as such in
> the bug, perhaps in order not to discourage the community from
> submitting them. What would be really nice would be some kind of tag or
> marker that automatically disappears as soon as a bug is changed in any
> way so that it gets the attention of a triager again.
>
> So our options are: 1) maintain state outside of the bugs themselves, 2)
> mark the "cannot be worked on state" inside the bug using a tag, or 3)
> be forever plagued with bugs in our triage lists
>
>
> 2. Automatic submission of package maintainer script failures
>
> 2a. Inadequate bug description often makes these very difficult to
> triage, and they stay on the list. Often the description is so vague
> that it's difficult to even prompt for further information without
> turning the bug into a full one-on-one support channel which we don't
> really have the bandwidth to do. I propose that these bugs are managed
> separately from regular triage, and/or have a standard "inadequate
> report" type standard response, but written specifically for maintainer
> script failures (since the usual one doesn't seem very appropriate for
> these cases).
>
> 2b. The reporters of most of these bugs with inadequate descriptions
> appear not to be interested in taking them further.
>
> 2c. Most are support requests and not bugs, reducing the time that
> triagers get to triage actual bugs.
>
> 2d. Desktop package maintainer scripts are expected rarely to fail, and
> if they do it probably is a bug. Server package maintainer scripts often
> try to set up daemons, but users customise daemons beyond what
> maintainer scripts can be expected to understand, causing them to fail
> and encourage reporting as bugs. And desktop users often install server
> packages which then confuses the matter when they try and upgrade.
> Linked with 2c, this causes a big problem for bug triage.
>
> Of course, there is no real distinction between server and desktop
> packages expect in the nature of what they are expected to do, and thus
> what their maintainer scripts try to do.
>
> I think it would be appropriate to see these bugs dealt with
> automatically (boilerplate response) unless or until:
>
> 1) It's a "proper" bug report where the submitter has actually spent
> some effort in reporting it (ie. with a bug description on par with
> what we'd expect for a manually submitted bug).
>
> 2) Aggregrate analysis suggests that the bug is real and thus we
> should look into it.
>
> 3) We've caught up with all the other bugs.
>
> At this point, I suggest always changing the subject of the bug from the
> automatic subject line to one that describes the real problem. This
> would really help to make the real bugs stand out from the automatic
> ones.
>
> The boilerplate would of course explain the situation with instructions
> on how to get the bug looked at if the reporter wants to take it further
> (eg. a variant of
> https://wiki.ubuntu.com/Bugs/Responses#Not_described_well)
>
> Robie
>
> --
> ubuntu-server mailing list
> ubuntu-server@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
> More info: https://wiki.ubuntu.com/ServerTeam
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
05-07-2012, 10:44 PM
Scott Kitterman
UDS-Q Topic: Server Bug Triage Process
On Monday, May 07, 2012 11:41:46 AM Robie Basak wrote:
> So our options are: 1) maintain state outside of the bugs themselves, 2)
> mark the "cannot be worked on state" inside the bug using a tag, or 3)
> be forever plagued with bugs in our triage lists
4) Go back to the way it used to be (no auto expiring of incomplete) and mark
such bugs as incomplete. Currently incomplete mostly means "I'm marking your
bug invalid, but I'm not willing to tell you - I'll let LP do it after it
times out".
Scott K
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
UDS-Q Topic: Server Bug Triage Process
