FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Kernel Team

 
 
LinkBack Thread Tools
 
Old 10-28-2008, 06:20 PM
Stefan Bader
 
Default RFC: Stable kernel updates and the SRU process

Martin Pitt wrote:
> Stefan Bader [2008-10-28 12:44 -0400]:
>> Still I think it would be better to revert selectively if there are regressions
>> found than only to select a few patches and miss fixes until we get bitten by them.
>
> That's not actually that preferable. It's generally better to keep
> known bugs which people got used to than breaking something that
> worked before (which is generally a disaster in critical environments,
> whereas bugs which have been there forever are merely an
> inconvenience). That obviously doesn't apply to things like the recent
> e1000 NVRAM incident, but for most severity "normal" bugs.
>
> Martin
>

This definitely should be accompanied by a wider or longer testing period. I
fully agree with that. My feeling would be that changes to the upstream stable
kernel are pretty well like SRUs in the way that they normally fix something to
get included. I don't have numbers regarding how many times the kernel broke
because of a stable patch. To minimize the risk there we could delay the import
of the next stable patchset a few weeks as well as enlarge the time an update
has to remain in -proposed.


--

When all other means of communication fail, try words!



--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-28-2008, 08:50 PM
Pete Graner
 
Default RFC: Stable kernel updates and the SRU process

Stefan Bader wrote:
> Martin Pitt wrote:
>> Stefan Bader [2008-10-28 12:44 -0400]:
>>> Still I think it would be better to revert selectively if there are regressions
>>> found than only to select a few patches and miss fixes until we get bitten by them.
>> That's not actually that preferable. It's generally better to keep
>> known bugs which people got used to than breaking something that
>> worked before (which is generally a disaster in critical environments,
>> whereas bugs which have been there forever are merely an
>> inconvenience). That obviously doesn't apply to things like the recent
>> e1000 NVRAM incident, but for most severity "normal" bugs.
>>
>> Martin
>>
>
> This definitely should be accompanied by a wider or longer testing period. I
> fully agree with that. My feeling would be that changes to the upstream stable
> kernel are pretty well like SRUs in the way that they normally fix something to
> get included. I don't have numbers regarding how many times the kernel broke
> because of a stable patch. To minimize the risk there we could delay the import
> of the next stable patchset a few weeks as well as enlarge the time an update
> has to remain in -proposed.
>
>

Does upstream have this documented anywhere? I'm not comfortable
adopting a process based on someone elses that is not documented,
followed and enforced.

Additionally I would support this if we can get this tested across all
the hardware we have. This ensures we can boot and pass a functions
check of the major sub-systems.

cc'ing in heno for comment. heno you can find the full thread here to
get up to speed:
https://lists.ubuntu.com/archives/kernel-team/2008-October/003364.html


~pete

--
Pete Graner <pgraner@canonical.com>

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-29-2008, 12:01 AM
Henrik Nilsen Omma
 
Default RFC: Stable kernel updates and the SRU process

Pete Graner wrote:

> Does upstream have this documented anywhere? I'm not comfortable
> adopting a process based on someone elses that is not documented,
> followed and enforced.
>
> Additionally I would support this if we can get this tested across all
> the hardware we have. This ensures we can boot and pass a functions
> check of the major sub-systems.
>
> cc'ing in heno for comment. heno you can find the full thread here to
> get up to speed:
> https://lists.ubuntu.com/archives/kernel-team/2008-October/003364.html

I generally support the proposed change provided that the SRU process be
replaced with another rigorous QA process. It should focus on regression
testing and broad hardware coverage.

We can definitely run automated tests across the hardware in our labs,
but I would appreciate some help from the kernel team in extending our
test suite for this. We currently run a fairly basic set of tests to
boot, install and check for basic HW support. We are also equiped to run
3rd party suites like autotest and ltp but we don't have the right
post-processing set up for these. I think we should cherry-pick a
selection of tests from these suites to run on each kernel version upgrade.

I'd also like for us to make these tests available to a wider audience
so we can encourage users to test on a greater variety of HW while the
new kernel is in -proposed. We should set up a tracking page to monitor
feedback from users for each .y update (could be an LP, list or based on
the SRU tracker or ISO tracker)

A QA procedure strawman:

* 2 weeks minimum in proposed and new uploads on the same .y cycle zeros
the clock

* We need to be able to identify blockers for each .y release. I propose
we set up a new project (ubuntu-kernel-updates?) that will have
milestones like intrepid-update-1. We can then set a task and milestone
for bugs reported during -proposed testing. High and Critical bugs
should be regarded as release-blockers.

* There should be explicit sign-off from QA before a new .y update goes
out, analogous to the role the SRU team plays now. I suggest Steve and
Leann should form a board that would make that call.

Henrik

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-29-2008, 08:27 AM
"Amit Kucheria"
 
Default RFC: Stable kernel updates and the SRU process

On Tue, Oct 28, 2008 at 11:50 PM, Pete Graner <pgraner@canonical.com> wrote:

> Does upstream have this documented anywhere? I'm not comfortable
> adopting a process based on someone elses that is not documented,
> followed and enforced.

This is the best thread I could find related to discussion of the
stable tree process: http://lwn.net/Articles/126915/

And the results of the discussion were documented in the kernel at
Documentation/stable_kernel_rules.txt

> Additionally I would support this if we can get this tested across all
> the hardware we have. This ensures we can boot and pass a functions
> check of the major sub-systems.

In order to do this, we would have to participate (and possibly urge
customers?) in the upstream stable review process. So we roll out a
-proposed kernel with all the proposed patches and report back to the
stable ML on issues.

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-29-2008, 12:54 PM
Tim Gardner
 
Default RFC: Stable kernel updates and the SRU process

Ben Collins wrote:
> Stefan Bader wrote:
>> Tim Gardner wrote:
>>> I would like to propose the following change in policy with regard to
>>> the Ubuntu kernel and stable tree updates.
>>>
>>> In the past our kernels have incorporated stable kernel tree updates up
>>> to the point of kernel freeze. After that point, only those patches that
>>> were specifically referenced by SRU Launchpad reports were applied.
>>>
>>> The upstream process for stable tree updates is quite similar in scope
>>> to the SRU process, e.g., each patch has to demonstrably fix a bug, and
>>> each patch is vetted by upstream by originating either directly from
>>> Linus' tree or in a minimally backported form of that patch.
>>>
>>> I think we are remiss if we do not take advantage of that upstream
>>> process to improve our kernel. Therefore I propose that we modify our
>>> kernel update policy such that we adopt stable kernel updates at
>>> appropriate points in the release process (avoiding kernel freezes etc)
>>> for as long as upstream continues to provide updates, and that these
>>> stable kernel updates not be subject to the SRU process.
>>>
>>> Comments?
>>>
>>> rtg
>>
>> I would also be in favor of such a change. In the past there have been
>> several
>> bug that were fixed by changes in the stable tree which we had to
>> search for
>> and then pull in. Also the patches in the stable tree are generally
>> small and
>> it takes a review process to get them in. So why not simplify our
>> workload by
>> pulling them in by default. I'd also only do one tracking bug per
>> stable release.
>>
>
> I should point out that regardless of the upstream process, we have, in
> the past, gotten regressions due to these changes.
>
> Hence the reason we have the current policy.
>

I think the odds of any particular stable update causing a regression
are no different then the average SRU patch because typically we won't
accept an SRU patch of any complexity unless it originates from
upstream, the source of all stable patches anyway.

rtg
--
Tim Gardner tim.gardner@canonical.com

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-29-2008, 01:26 PM
Tim Gardner
 
Default RFC: Stable kernel updates and the SRU process

Henrik Nilsen Omma wrote:
>
> I generally support the proposed change provided that the SRU process be
> replaced with another rigorous QA process. It should focus on regression
> testing and broad hardware coverage.
>
> We can definitely run automated tests across the hardware in our labs,
> but I would appreciate some help from the kernel team in extending our
> test suite for this. We currently run a fairly basic set of tests to
> boot, install and check for basic HW support. We are also equiped to run
> 3rd party suites like autotest and ltp but we don't have the right
> post-processing set up for these. I think we should cherry-pick a
> selection of tests from these suites to run on each kernel version upgrade.
>
> I'd also like for us to make these tests available to a wider audience
> so we can encourage users to test on a greater variety of HW while the
> new kernel is in -proposed. We should set up a tracking page to monitor
> feedback from users for each .y update (could be an LP, list or based on
> the SRU tracker or ISO tracker)
>
> A QA procedure strawman:
>
> * 2 weeks minimum in proposed and new uploads on the same .y cycle zeros
> the clock
>
> * We need to be able to identify blockers for each .y release. I propose
> we set up a new project (ubuntu-kernel-updates?) that will have
> milestones like intrepid-update-1. We can then set a task and milestone
> for bugs reported during -proposed testing. High and Critical bugs
> should be regarded as release-blockers.
>
> * There should be explicit sign-off from QA before a new .y update goes
> out, analogous to the role the SRU team plays now. I suggest Steve and
> Leann should form a board that would make that call.
>
> Henrik
>

I didn't really intend to propose changing the SRU process. Is there any
reason that expanded hardware testing can't take advantage of the
existing SRU mechanism (which I consider to work pretty well) ?

Perhaps an alternative solution is to automate the submission of stable
patch SRUs, tracked as individual bug reports with their own regression
results. Or, we can just bite the bullet and submit the SRUs manually.
It'll only take a few tedious hours in most cases.

My real point is that there is much goodness in the stable updates and I
don't want our current process or policy to prevent us from taking
advantage of them.

rtg
--
Tim Gardner tim.gardner@canonical.com

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 10-29-2008, 02:28 PM
Ben Collins
 
Default RFC: Stable kernel updates and the SRU process

Tim Gardner wrote:
> Ben Collins wrote:
>> Stefan Bader wrote:
>>> Tim Gardner wrote:
>>>> I would like to propose the following change in policy with regard to
>>>> the Ubuntu kernel and stable tree updates.
>>>>
>>>> In the past our kernels have incorporated stable kernel tree updates up
>>>> to the point of kernel freeze. After that point, only those patches that
>>>> were specifically referenced by SRU Launchpad reports were applied.
>>>>
>>>> The upstream process for stable tree updates is quite similar in scope
>>>> to the SRU process, e.g., each patch has to demonstrably fix a bug, and
>>>> each patch is vetted by upstream by originating either directly from
>>>> Linus' tree or in a minimally backported form of that patch.
>>>>
>>>> I think we are remiss if we do not take advantage of that upstream
>>>> process to improve our kernel. Therefore I propose that we modify our
>>>> kernel update policy such that we adopt stable kernel updates at
>>>> appropriate points in the release process (avoiding kernel freezes etc)
>>>> for as long as upstream continues to provide updates, and that these
>>>> stable kernel updates not be subject to the SRU process.
>>>>
>>>> Comments?
>>>>
>>>> rtg
>>> I would also be in favor of such a change. In the past there have been
>>> several
>>> bug that were fixed by changes in the stable tree which we had to
>>> search for
>>> and then pull in. Also the patches in the stable tree are generally
>>> small and
>>> it takes a review process to get them in. So why not simplify our
>>> workload by
>>> pulling them in by default. I'd also only do one tracking bug per
>>> stable release.
>>>
>> I should point out that regardless of the upstream process, we have, in
>> the past, gotten regressions due to these changes.
>>
>> Hence the reason we have the current policy.
>>
>
> I think the odds of any particular stable update causing a regression
> are no different then the average SRU patch because typically we won't
> accept an SRU patch of any complexity unless it originates from
> upstream, the source of all stable patches anyway.
>

True, but the odds of a set of updates causing a regression increases
the more patches you add to the set.

Plus, when we cherry pick from .y for SRU's, we generally eyeball the
patches individually, which wont be done when a direct merge with .y.

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 11-28-2008, 04:02 PM
Matt Zimmerman
 
Default RFC: Stable kernel updates and the SRU process

On Wed, Oct 29, 2008 at 01:01:00AM +0000, Henrik Nilsen Omma wrote:
> I generally support the proposed change provided that the SRU process be
> replaced with another rigorous QA process. It should focus on regression
> testing and broad hardware coverage.
>
> We can definitely run automated tests across the hardware in our labs,
> but I would appreciate some help from the kernel team in extending our
> test suite for this. We currently run a fairly basic set of tests to
> boot, install and check for basic HW support. We are also equiped to run
> 3rd party suites like autotest and ltp but we don't have the right
> post-processing set up for these. I think we should cherry-pick a
> selection of tests from these suites to run on each kernel version
> upgrade.

I think you could probably get a lot of mileage out of a test suite which
was focused on trying to identify regressions, rather than verify
functionality.

For example, it could confirm that the devices were still bound to the same
drivers, that there were no unexpected changes to dmesg output, that all of
the same devices are detected, etc.

--
- mdz

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 

Thread Tools




All times are GMT. The time now is 08:38 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org