Kernel Bug Migration
 

Hi Guys,

As most of you know, beginning with the Hardy release cycle the Ubuntu
kernel source package naming convention changed from linux-source-2.6.xx
to just linux. I think it is important that as we move forward we be
sure to migrate any bugs open against older kernels to the most recent
kernel. Otherwise, I fear bugs may stagnate and go unaddressed. There
is a significant amount of bugs open against older Ubuntu kernels.
Manually processing all these bugs by hand would be inefficient. I'd
like to propose automating the process of moving some of these bugs
forward. I've documented the process, criteria, and stock replies I'd
like to use at the following wiki:

https://wiki.ubuntu.com/QATeam/KernelBugMigration

I'll inline portions of the wiki below.

Criteria:
1. Bug is not undergoing the SRU process (ubuntu-sru is not a
subscriber and the bug is not specifically targeted to a
release)
2. Bug is not a security bug (security_related is true)
3. Bug has less than or equal to (<=) 10 subscribers
4. Bug must be against a linux-source-2.6.xx package and have a
Status of either New, Incomplete, Confirmed, or Triaged
5. Bug must only be tasked to a single linux-source-2.6.xx package
(ie bugs against multiple kernel releases will not be handled in
this first pass)
6. Bug does not already have a "linux" task (just reiterating
above)
7. Bug is not a High or Critical bug - we should look at the High
and Critical bugs manually
8. Bug is not In Progress or Fix Committed - these should be looked
at manually too as they may really be fixed

Process:
1. Launchpad janitor will automatically post a stock reply (see
wiki) depending on which kernel is being processed
2. linux-source-2.6.15 bugs will be renamed to the new "linux"
kernel package. Also, tag these 'linux-source-2.6.15' to
maintain package history
3. linux-source-2.6.17 bugs will be closed as Won't Fix as the 18
month support period has ended. The new "linux" task will be
added but marked as Incomplete. See stock reply for more info.
Also, tag these 'linux-source-2.6.17' to easily list which bugs
were migrated and isolate possible candidates to close if we've
received no response for testing with the latest release.
4. linux-source-2.6.20 and linux-source-2.6.22 bugs will be marked
as Won't Fix. The new "linux" task will be added but marked as
Incomplete. See stock reply for more info. Also, tag these
'linux-source-2.6.20' or 'linux-source-2.6.22' to easily list
which bugs were migrated and isolate possible candidates to
close if we've received no response for testing with the latest
release.

I'd appreciate any feedback you guys may have. Ideally I'd like to
start moving forward with this beginning in July.

Thanks,
Leann


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

Leann Ogasawara wrote:
> Hi Guys,
>
> As most of you know, beginning with the Hardy release cycle the Ubuntu
> kernel source package naming convention changed from linux-source-2.6.xx
> to just linux. I think it is important that as we move forward we be
> sure to migrate any bugs open against older kernels to the most recent
> kernel. Otherwise, I fear bugs may stagnate and go unaddressed. There
> is a significant amount of bugs open against older Ubuntu kernels.
> Manually processing all these bugs by hand would be inefficient. I'd
> like to propose automating the process of moving some of these bugs
> forward. I've documented the process, criteria, and stock replies I'd
> like to use at the following wiki:
>
> https://wiki.ubuntu.com/QATeam/KernelBugMigration
>
> I'll inline portions of the wiki below.
>
> Criteria:
> 1. Bug is not undergoing the SRU process (ubuntu-sru is not a
> subscriber and the bug is not specifically targeted to a
> release)
> 2. Bug is not a security bug (security_related is true)
> 3. Bug has less than or equal to (<=) 10 subscribers
> 4. Bug must be against a linux-source-2.6.xx package and have a
> Status of either New, Incomplete, Confirmed, or Triaged
> 5. Bug must only be tasked to a single linux-source-2.6.xx package
> (ie bugs against multiple kernel releases will not be handled in
> this first pass)
> 6. Bug does not already have a "linux" task (just reiterating
> above)
> 7. Bug is not a High or Critical bug - we should look at the High
> and Critical bugs manually
> 8. Bug is not In Progress or Fix Committed - these should be looked
> at manually too as they may really be fixed
>
> Process:
> 1. Launchpad janitor will automatically post a stock reply (see
> wiki) depending on which kernel is being processed
> 2. linux-source-2.6.15 bugs will be renamed to the new "linux"
> kernel package. Also, tag these 'linux-source-2.6.15' to
> maintain package history
> 3. linux-source-2.6.17 bugs will be closed as Won't Fix as the 18
> month support period has ended. The new "linux" task will be
> added but marked as Incomplete. See stock reply for more info.
> Also, tag these 'linux-source-2.6.17' to easily list which bugs
> were migrated and isolate possible candidates to close if we've
> received no response for testing with the latest release.
> 4. linux-source-2.6.20 and linux-source-2.6.22 bugs will be marked
> as Won't Fix. The new "linux" task will be added but marked as
> Incomplete. See stock reply for more info. Also, tag these
> 'linux-source-2.6.20' or 'linux-source-2.6.22' to easily list
> which bugs were migrated and isolate possible candidates to
> close if we've received no response for testing with the latest
> release.
>
> I'd appreciate any feedback you guys may have. Ideally I'd like to
> start moving forward with this beginning in July.
>
> Thanks,
> Leann
>
>

If I understand your algorithm correctly, you plan to migrate unpopular,
little noticed bugs from older releases. I'm wondering, why bother? In
all honesty, that class of bugs will never get any attention, so why not
just close them as "Won't fix" ? I'm much more interested in fixing the
current version issues because that is where my efforts have the most
bang for the buck. I'm far more likely to be able to find an upstream
patch for a _recent_ release of the kernel or driver, and its rare that
the kernel team has the time or hardware resources to develop a patch
from scratch (which is what we would have to do in most cases for older
releases). Its simply not cost effective to fix a bug on an older
release that affect a very few people when I can be working on one that
affects many more.

It appears to me that migrating these bugs to a common package will make
it more difficult for me to weed through the bug list in order to decide
what to work on.

rtg

--
Tim Gardner tim.gardner@ubuntu.com

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

On Tue, 2008-06-10 at 10:20 +0100, Tim Gardner wrote:
[snip]
> If I understand your algorithm correctly, you plan to migrate unpopular,
> little noticed bugs from older releases. I'm wondering, why bother?

I'm not sure I'd classify them as "unpopular" but I definitely agree
they go unnoticed. That's why I'd at least like them to test with the
most recent kernel to verify if their bug has been fixed or not. If
not, it's likely the issue may exist upstream I imagine so this may be a
scenario to forward bugs upstream.

> In all honesty, that class of bugs will never get any attention, so why not
> just close them as "Won't fix" ?

At least for the 2.6.17, 2.6.20, and 2.6.22 bugs we are closing them as
"Won't Fix" with a request to test the latest release. We're then
automatically opening and setting the linux task to Incomplete until
they test and provide feedback. So these should stay off your radar
until they're actually tested and confirmed to exist against the latest
release.

For 2.6.15 bugs I proposed a slightly different process just because it
was an LTS release. But I'd be fine with using the same procedure above
that I'd use with the other non-LTS kernels.

> I'm much more interested in fixing the
> current version issues because that is where my efforts have the most
> bang for the buck. I'm far more likely to be able to find an upstream
> patch for a _recent_ release of the kernel or driver, and its rare that
> the kernel team has the time or hardware resources to develop a patch
> from scratch (which is what we would have to do in most cases for older
> releases). Its simply not cost effective to fix a bug on an older
> release that affect a very few people when I can be working on one that
> affects many more.

Agreed. I know your guys' time is best spent focusing on the bugs which
affect the current release. However, I don't think that means we should
just ignore legitimate kernel bugs reported against older kernel
releases but may still exist in the current one.

> It appears to me that migrating these bugs to a common package will make
> it more difficult for me to weed through the bug list in order to decide
> what to work on.

Hopefully by setting the bugs to "Won't Fix" for the older kernels and
"Incomplete" for the recent kernel, they should stay off your radar
until they are confirmed to exist in the latest release.

Thoughts on this?

Thanks for the feedback already,
Leann


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

Leann Ogasawara wrote:
> On Tue, 2008-06-10 at 10:20 +0100, Tim Gardner wrote:
> [snip]
>> If I understand your algorithm correctly, you plan to migrate unpopular,
>> little noticed bugs from older releases. I'm wondering, why bother?
>
> I'm not sure I'd classify them as "unpopular" but I definitely agree
> they go unnoticed. That's why I'd at least like them to test with the
> most recent kernel to verify if their bug has been fixed or not. If
> not, it's likely the issue may exist upstream I imagine so this may be a
> scenario to forward bugs upstream.
>

I meant unpopular in the sense that your algorithm requires that this
class of bugs have fewer then 10 comments.

>> In all honesty, that class of bugs will never get any attention, so why not
>> just close them as "Won't fix" ?
>
> At least for the 2.6.17, 2.6.20, and 2.6.22 bugs we are closing them as
> "Won't Fix" with a request to test the latest release. We're then
> automatically opening and setting the linux task to Incomplete until
> they test and provide feedback. So these should stay off your radar
> until they're actually tested and confirmed to exist against the latest
> release.
>
> For 2.6.15 bugs I proposed a slightly different process just because it
> was an LTS release. But I'd be fine with using the same procedure above
> that I'd use with the other non-LTS kernels.
>

I can't think of a good reason not to use the same algorithm for Dapper.
My support focus is primarily on server and CVE issues. Desktop users
suffering from bugs are just going to have to upgrade to a recent
release where their issue is more likely to get addressed.

>> I'm much more interested in fixing the
>> current version issues because that is where my efforts have the most
>> bang for the buck. I'm far more likely to be able to find an upstream
>> patch for a _recent_ release of the kernel or driver, and its rare that
>> the kernel team has the time or hardware resources to develop a patch
>> from scratch (which is what we would have to do in most cases for older
>> releases). Its simply not cost effective to fix a bug on an older
>> release that affect a very few people when I can be working on one that
>> affects many more.
>
> Agreed. I know your guys' time is best spent focusing on the bugs which
> affect the current release. However, I don't think that means we should
> just ignore legitimate kernel bugs reported against older kernel
> releases but may still exist in the current one.
>
>> It appears to me that migrating these bugs to a common package will make
>> it more difficult for me to weed through the bug list in order to decide
>> what to work on.
>
> Hopefully by setting the bugs to "Won't Fix" for the older kernels and
> "Incomplete" for the recent kernel, they should stay off your radar
> until they are confirmed to exist in the latest release.
>
> Thoughts on this?
>
> Thanks for the feedback already,
> Leann
>

I think that if this helps your ability to triage and categorize bugs,
then you should do what you need to, as long as you keep producing the
Recommended Bug List. In practice, your recommended bug list is about
the only thing I (or any of the kernel team members) look at when
deciding what to work on.

rtg
--
Tim Gardner tim.gardner@ubuntu.com

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

On Wed, 2008-06-11 at 17:02 +0100, Tim Gardner wrote:
> Leann Ogasawara wrote:
> > On Tue, 2008-06-10 at 10:20 +0100, Tim Gardner wrote:
> > [snip]
> >> If I understand your algorithm correctly, you plan to migrate unpopular,
> >> little noticed bugs from older releases. I'm wondering, why bother?
> >
> > I'm not sure I'd classify them as "unpopular" but I definitely agree
> > they go unnoticed. That's why I'd at least like them to test with the
> > most recent kernel to verify if their bug has been fixed or not. If
> > not, it's likely the issue may exist upstream I imagine so this may be a
> > scenario to forward bugs upstream.
> >
>
> I meant unpopular in the sense that your algorithm requires that this
> class of bugs have fewer then 10 comments.

Ah yes, the <= 10 subscribers. I think I'm going to get rid of this
criteria. I'll manually take a peek at the ones with a high subscriber
count and we'll clean up the rest automatically.

> > For 2.6.15 bugs I proposed a slightly different process just because it
> > was an LTS release. But I'd be fine with using the same procedure above
> > that I'd use with the other non-LTS kernels.
> >
>
> I can't think of a good reason not to use the same algorithm for Dapper.
> My support focus is primarily on server and CVE issues. Desktop users
> suffering from bugs are just going to have to upgrade to a recent
> release where their issue is more likely to get addressed.

Agreed. I'll use the same algorithm for all the kernels then.

[snip]
> I think that if this helps your ability to triage and categorize bugs,
> then you should do what you need to, as long as you keep producing the
> Recommended Bug List. In practice, your recommended bug list is about
> the only thing I (or any of the kernel team members) look at when
> deciding what to work on.

Yup, I'll definitely keep sending out the recommended bug list. Aside
from some extra bug mail you'll probably see, I'm hoping this won't
produce any major distractions or interruptions for you guys.

Thanks,
Leann



--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

2008/6/11 Tim Gardner <tim.gardner@canonical.com>:
> Leann Ogasawara wrote:
>> For 2.6.15 bugs I proposed a slightly different process just because it
>> was an LTS release. But I'd be fine with using the same procedure above
>> that I'd use with the other non-LTS kernels.
>>
>
> I can't think of a good reason not to use the same algorithm for Dapper.
> My support focus is primarily on server and CVE issues. Desktop users
> suffering from bugs are just going to have to upgrade to a recent
> release where their issue is more likely to get addressed.

Seems like the opposite of the whole LTS idea. Fix-by-upgrade is not a
solution. Newer release could introduce other bugs and then what?
Upgrade to even newer version? Not to mention that LTS releases are
rare, so there might be no newer LTS version to upgrade to.

IMO bugs should be fixed in LTS kernels and/or newer kernels should be
backported to LTS releases.

--

Krzysztof Lichota

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

On Tue, 2008-06-17 at 10:55 +0200, Krzysztof Lichota wrote:
> 2008/6/11 Tim Gardner <tim.gardner@canonical.com>:
> > Leann Ogasawara wrote:
> >> For 2.6.15 bugs I proposed a slightly different process just because it
> >> was an LTS release. But I'd be fine with using the same procedure above
> >> that I'd use with the other non-LTS kernels.
> >>
> >
> > I can't think of a good reason not to use the same algorithm for Dapper.
> > My support focus is primarily on server and CVE issues. Desktop users
> > suffering from bugs are just going to have to upgrade to a recent
> > release where their issue is more likely to get addressed.
>
> Seems like the opposite of the whole LTS idea. Fix-by-upgrade is not a
> solution. Newer release could introduce other bugs and then what?
> Upgrade to even newer version? Not to mention that LTS releases are
> rare, so there might be no newer LTS version to upgrade to.
>
> IMO bugs should be fixed in LTS kernels and/or newer kernels should be
> backported to LTS releases.

For LTS releases, we only guarantee that we wont regress the kernel, and
that we will keep it up-to-date with security fixes. Fixing known bugs
is on a best-effort basis and is not guaranteed. So this isn't anything
new.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

2008/6/17 Ben Collins <ben.collins@canonical.com>:
>> Seems like the opposite of the whole LTS idea. Fix-by-upgrade is not a
>> solution. Newer release could introduce other bugs and then what?
>> Upgrade to even newer version? Not to mention that LTS releases are
>> rare, so there might be no newer LTS version to upgrade to.
>>
>> IMO bugs should be fixed in LTS kernels and/or newer kernels should be
>> backported to LTS releases.
>
> For LTS releases, we only guarantee that we wont regress the kernel, and
> that we will keep it up-to-date with security fixes. Fixing known bugs
> is on a best-effort basis and is not guaranteed. So this isn't anything
> new.

So it comes down to: there is no use in reporting bugs against LTS as
they will be fixed in 2 years?

Shouldn't it be "we will try hard to iron out all bugs for LTS and
deliver it in non-disruptible, voluntary manner (through -backports or
-updates)"?

--

Krzysztof Lichota

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

On Tue, 2008-06-17 at 16:18 +0200, Krzysztof Lichota wrote:
> 2008/6/17 Ben Collins <ben.collins@canonical.com>:
> >> Seems like the opposite of the whole LTS idea. Fix-by-upgrade is not a
> >> solution. Newer release could introduce other bugs and then what?
> >> Upgrade to even newer version? Not to mention that LTS releases are
> >> rare, so there might be no newer LTS version to upgrade to.
> >>
> >> IMO bugs should be fixed in LTS kernels and/or newer kernels should be
> >> backported to LTS releases.
> >
> > For LTS releases, we only guarantee that we wont regress the kernel, and
> > that we will keep it up-to-date with security fixes. Fixing known bugs
> > is on a best-effort basis and is not guaranteed. So this isn't anything
> > new.
>
> So it comes down to: there is no use in reporting bugs against LTS as
> they will be fixed in 2 years?

No, since those bugs help us to fix things immediately, not in 2 years.

> Shouldn't it be "we will try hard to iron out all bugs for LTS and
> deliver it in non-disruptible, voluntary manner (through -backports or
> -updates)"?

It really depends on the extent of those bugs. If it affects a lot of
people, then that's where the "best effort" comes in. We obviously
aren't going to fix everyone's bugs or bugs that only affect a minority
of users because we have to weigh our efforts and the possibilities for
regressions.

The nature of the bugs also comes into play. If it's an infrequent
annoyance, then we may not fix it either. Where as, if it keeps people
from using the OS at all, then it is more likely to get fixed.

So let's not try to put a blanket policy around all bugs affecting an
LTS. There are a lot of criteria involved.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Kernel Bug Migration
 

2008/6/17 Ben Collins <ben.collins@canonical.com>:
>> Shouldn't it be "we will try hard to iron out all bugs for LTS and
>> deliver it in non-disruptible, voluntary manner (through -backports or
>> -updates)"?
>
> It really depends on the extent of those bugs. If it affects a lot of
> people, then that's where the "best effort" comes in. We obviously
> aren't going to fix everyone's bugs or bugs that only affect a minority
> of users because we have to weigh our efforts and the possibilities for
> regressions.
>
> The nature of the bugs also comes into play. If it's an infrequent
> annoyance, then we may not fix it either. Where as, if it keeps people
> from using the OS at all, then it is more likely to get fixed.
>
> So let's not try to put a blanket policy around all bugs affecting an
> LTS. There are a lot of criteria involved.

OK. Thanks for explanation.

--

Krzysztof Lichota

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team