Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Development (http://www.linux-archive.org/fedora-development/)
-   -   The new Update Acceptance Criteria are broken (http://www.linux-archive.org/fedora-development/446330-new-update-acceptance-criteria-broken.html)

"Clyde E. Kunkel" 10-31-2010 01:16 PM

The new Update Acceptance Criteria are broken
 
On 10/31/2010 03:18 AM, Michael Schwendt wrote:
> On Sun, 31 Oct 2010 04:37:38 +0100, Kevin wrote:
>
>> Martin Stransky wrote:
>>> there's a new Firefox update waiting in Bodhi and we can't push it to
>>> stable because of new rules. We recommend you to update to it ASAP as it
>>> fixes a public critical 0day vulnerability
>>> (https://bugzilla.mozilla.org/show_bug.cgi?id=607222).
>>
>> Looks like the F13 build got karma quickly enough to land directly in stable
>> after all, the F12 build, on the other hand, was stuck in testing for 2 days
>> before finally making it out to stable. Yet another blatant example of
>> failure of the Update Acceptance Criteria, needlessly exposing our users to
>> critical vulnerabilities.
>>
>> (And no, by giving yet another special exception to Firefox wouldn't be a
>> solution. ;-) This problem can hit any other app as well.)
>>
>> Kevin Kofler
>
> Okay, feedback time.
>
> Lately, there have been several attempts at urging proventesters (and not
> just testers in general) to give positive karma for aging critpath updates.
> It also has been decided by someone (or maybe even a comittee) to spam
> proventesters daily with "[old_testing_critpath]" messages for all three
> dist releases, with no day to unsubscribe from that (other than leaving
> proventesters group, which is what at least one person has threatened with,
> or filtering those messages).
>
> Dunno about other testers (and there aren't many yet), but I have abandoned
> F-12 long ago due to lack of time when F-13 became the one to use on a daily
> basis. And some time before F-14 Beta, my desktop has been switched to boot
> F-14 by default. That's the only opportunity to evaluate F-14 early and
> possibly find issues prior to its release. On the contrary, most of Fedora's
> users will wait for the final release, and many users will wait even longer.
> It's highly likely that bugzilla can confirm that.
>
> F-14 is the the only way forward, and don't like to spend time on F-13 and
> older anymore. That also applies to packagers I maintain or monitor. I simply
> don't see the user base [target group] anymore.
>
> About positive karma in bodhi, I don't feel comfortable signing off
> arbitrary updates just because they didn't crash for me after five
> minutes. With some updates, regression has slipped through already.
> And the more bugs an update addresses with either patches or a version
> upgrade, the more careful I would like to be when testing something.
> Also, in my book, an update working on F-14 may still malfunction on an
> older dist release due to differences in dependences and the core setup. I
> still don't understand why some non-security updates are rushed out with
> sometimes not even the package maintainer(s) having tested them at all.

I am willing to work with the older, still supported, distros, but would
really appreciate test cases since most of the critical-path bugs the
update addresses are not common and I haven't run into them. That said,
if the update remains without karma, the release is within a month of
end-of-life, then the update could be left in updates testing and docs
changed to provide a warning. I don't think there would be that much
impact on storage to keep an updates-testing repo around on the mirrors
that choose to provide the release. Most just delete the release anyway.

Regards,
OldFart
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Kevin Fenzi 11-01-2010 02:35 PM

The new Update Acceptance Criteria are broken
 
On Sun, 31 Oct 2010 10:16:41 -0400
"Clyde E. Kunkel" <clydekunkel7734@cox.net> wrote:

> On 10/31/2010 03:18 AM, Michael Schwendt wrote:
> > Okay, feedback time.
> >
> > Lately, there have been several attempts at urging proventesters
> > (and not just testers in general) to give positive karma for aging
> > critpath updates. It also has been decided by someone (or maybe
> > even a comittee) to spam proventesters daily with
> > "[old_testing_critpath]" messages for all three dist releases, with
> > no day to unsubscribe from that (other than leaving proventesters
> > group, which is what at least one person has threatened with, or
> > filtering those messages).

Yeah, I am not sure at all how usefull those emails are.
There are a variety of ways to see what things need testing, sending
emails to proventesters a bunch isn't likely to be a very nice one.

> > Dunno about other testers (and there aren't many yet), but I have
> > abandoned F-12 long ago due to lack of time when F-13 became the
> > one to use on a daily basis. And some time before F-14 Beta, my
> > desktop has been switched to boot F-14 by default. That's the only
> > opportunity to evaluate F-14 early and possibly find issues prior
> > to its release. On the contrary, most of Fedora's users will wait
> > for the final release, and many users will wait even longer. It's
> > highly likely that bugzilla can confirm that.

I've got a f12 vm that I can run command line testing easily and some
limited gui testing (vnc).

> > F-14 is the the only way forward, and don't like to spend time on
> > F-13 and older anymore. That also applies to packagers I maintain
> > or monitor. I simply don't see the user base [target group] anymore.

It's really hard to tell I'm afraid, but I understand your feeling
there.

I also would love some simple per-package test plans. My thought was
that our testing could start out with a very low bar, and go from
there. ie, Does it install. Does it run and display a window, etc.
If there was a test plan page for better testing that would be great.
If there was a way to test a specific bug that would also be great.
(several updates have included test cases in their bug that were great
to test and confirm it was fixed).

Anyhow, I agree we should look at adjusting the f12 setup (although its
only got 1 month left), as well as look at dropping the emails to
proventesters with old testing stuff.

Thanks for the constructive feedback Clyde and Michael.

Specific plans for changing things for the better welcome.

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Till Maas 11-12-2010 06:03 PM

The new Update Acceptance Criteria are broken
 
On Mon, Nov 01, 2010 at 10:09:17AM -0700, Adam Williamson wrote:

> I disagree. The evidence you cite does not support this conclusion. We
> implemented the policies for three releases. There are significant
> problems with one release. This does not justify the conclusion that the
> policies should be entirely repealed.

It was brought to my attention that also current Fedora releases have
problems with delaying important security updates. A fix for a remote
code execution vulnerability in proftpd was only pushed to stable with a
seven day delay:
https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc13
https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc14

And it is not a theoretical threat, I know that servers in the nearby
area have been exploited because of this vulnerability. Delaying such
updates seems to be a very bad idea. Even in the unlikely case that the
update was broken and made proftpd not start anymore, this is usually
not as bad as having the system corrupted by an evil attacker.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 11-12-2010 06:19 PM

The new Update Acceptance Criteria are broken
 
On Fri, 2010-11-12 at 20:03 +0100, Till Maas wrote:
> On Mon, Nov 01, 2010 at 10:09:17AM -0700, Adam Williamson wrote:
>
> > I disagree. The evidence you cite does not support this conclusion. We
> > implemented the policies for three releases. There are significant
> > problems with one release. This does not justify the conclusion that the
> > policies should be entirely repealed.
>
> It was brought to my attention that also current Fedora releases have
> problems with delaying important security updates. A fix for a remote
> code execution vulnerability in proftpd was only pushed to stable with a
> seven day delay:
> https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc13
> https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc14
>
> And it is not a theoretical threat, I know that servers in the nearby
> area have been exploited because of this vulnerability. Delaying such
> updates seems to be a very bad idea. Even in the unlikely case that the
> update was broken and made proftpd not start anymore, this is usually
> not as bad as having the system corrupted by an evil attacker.

Thanks for flagging this up.

I'm wondering if perhaps we should devise a system - maybe a sub-group
of proventesters - to ensure timely testing of security updates. wdyt?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Tom Lane 11-12-2010 06:32 PM

The new Update Acceptance Criteria are broken
 
Till Maas <opensource@till.name> writes:
> On Mon, Nov 01, 2010 at 10:09:17AM -0700, Adam Williamson wrote:
>> I disagree. The evidence you cite does not support this conclusion. We
>> implemented the policies for three releases. There are significant
>> problems with one release. This does not justify the conclusion that the
>> policies should be entirely repealed.

> It was brought to my attention that also current Fedora releases have
> problems with delaying important security updates.

Quite. In my little corner of the system, none of the last several
mysql and postgresql updates have gone out with less than a seven-day
delay, despite some of them being security updates (admittedly not
high-severity ones, but still). And the trend is downhill: out of the
last nine such updates, five shipped with zero karma because not even
one tester had got round to looking at them. How does it help anyone
to delay releases when no testing will happen?

It's absolutely crystal clear to me that we don't have enough tester
manpower to make the current policy workable; it's past time to stop
denying that. I'd suggest narrowing the policy to a small number of
critical packages, for which there might be some hope of it actually
working as designed.

regards, tom lane
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 11-12-2010 06:42 PM

The new Update Acceptance Criteria are broken
 
On Fri, 2010-11-12 at 14:32 -0500, Tom Lane wrote:
> Till Maas <opensource@till.name> writes:
> > On Mon, Nov 01, 2010 at 10:09:17AM -0700, Adam Williamson wrote:
> >> I disagree. The evidence you cite does not support this conclusion. We
> >> implemented the policies for three releases. There are significant
> >> problems with one release. This does not justify the conclusion that the
> >> policies should be entirely repealed.
>
> > It was brought to my attention that also current Fedora releases have
> > problems with delaying important security updates.
>
> Quite. In my little corner of the system, none of the last several
> mysql and postgresql updates have gone out with less than a seven-day
> delay, despite some of them being security updates (admittedly not
> high-severity ones, but still). And the trend is downhill: out of the
> last nine such updates, five shipped with zero karma because not even
> one tester had got round to looking at them. How does it help anyone
> to delay releases when no testing will happen?
>
> It's absolutely crystal clear to me that we don't have enough tester
> manpower to make the current policy workable; it's past time to stop
> denying that. I'd suggest narrowing the policy to a small number of
> critical packages, for which there might be some hope of it actually
> working as designed.

the policy is already differentiated between critpath and non-critpath
packages; critpath *require* testing, there's no 7-day clause.

it's worth noting that part of the point of the 7-day clause is to cover
'invisible testing'; even if people aren't posting feedback to Bodhi,
it's likely that if the update actually is broken, we will find out one
way or another within 7 days (some people will post negative feedback to
Bodhi but not positive; or we'll get notified on an ML, or forums, or
something).

it's also worth noting that this is a communal effort: we don't have a
big batch of testing robots who test whatever they're told to test.
(yet). It's going to work much better if developers take some
responsibility for getting their packages testing it. if you're
packaging something, presumably you know *somebody* who uses it: the
idea is that you can ask them to test it and provide the bodhi feedback,
not just rely on someone who runs fedora-easy-karma as a matter of
course providing feedback.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 11-12-2010 06:46 PM

The new Update Acceptance Criteria are broken
 
On Fri, 2010-11-12 at 14:32 -0500, Tom Lane wrote:

> It's absolutely crystal clear to me that we don't have enough tester
> manpower to make the current policy workable; it's past time to stop
> denying that. I'd suggest narrowing the policy to a small number of
> critical packages, for which there might be some hope of it actually
> working as designed.

BTW, another point worth noting is that there is not actually a specific
rule against +1ing your own updates. It's 'frowned upon', I guess, but
actually I think it's probably workable to say it's fine for the
developer to +1 their own update in Bodhi: *as long as you actually have
tested it*. For non-critpath updates especially. The Bodhi system is
essentially an honor system anyway. So how I'd see this working is if it
becomes clear that some maintainer is gaming the system by just +1ing
everything they submit, even if it actually turns out to be broken, we
look at saying they can't +1 their own updates any more. But if you
actually are conscientiously testing your own updates, that's probably
worth a +1 in Bodhi, for me.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Simo Sorce 11-12-2010 06:54 PM

The new Update Acceptance Criteria are broken
 
On Fri, 12 Nov 2010 11:19:22 -0800
Adam Williamson <awilliam@redhat.com> wrote:

> On Fri, 2010-11-12 at 20:03 +0100, Till Maas wrote:
> > On Mon, Nov 01, 2010 at 10:09:17AM -0700, Adam Williamson wrote:
> >
> > > I disagree. The evidence you cite does not support this
> > > conclusion. We implemented the policies for three releases. There
> > > are significant problems with one release. This does not justify
> > > the conclusion that the policies should be entirely repealed.
> >
> > It was brought to my attention that also current Fedora releases
> > have problems with delaying important security updates. A fix for a
> > remote code execution vulnerability in proftpd was only pushed to
> > stable with a seven day delay:
> > https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc13
> > https://admin.fedoraproject.org/updates/proftpd-1.3.3c-1.fc14
> >
> > And it is not a theoretical threat, I know that servers in the
> > nearby area have been exploited because of this vulnerability.
> > Delaying such updates seems to be a very bad idea. Even in the
> > unlikely case that the update was broken and made proftpd not start
> > anymore, this is usually not as bad as having the system corrupted
> > by an evil attacker.
>
> Thanks for flagging this up.
>
> I'm wondering if perhaps we should devise a system - maybe a sub-group
> of proventesters - to ensure timely testing of security updates. wdyt?

Adam why should security updates wait at all ?
Do you fear some packager will flag as security updates that are not ?
Surely we can deal with such maintainer if that happens...

Simo.

--
Simo Sorce * Red Hat, Inc * New York
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 11-12-2010 07:02 PM

The new Update Acceptance Criteria are broken
 
On Fri, 2010-11-12 at 14:54 -0500, Simo Sorce wrote:

> Adam why should security updates wait at all ?
> Do you fear some packager will flag as security updates that are not ?
> Surely we can deal with such maintainer if that happens...

I don't have a hugely strong opinion either way, but the stated reason
by those who do is that security updates can be broken just like any
other. We don't have a magic 'infallible' switch on packagers which we
toggle only when they're building a security update. :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Simo Sorce 11-12-2010 07:13 PM

The new Update Acceptance Criteria are broken
 
On Fri, 12 Nov 2010 12:02:03 -0800
Adam Williamson <awilliam@redhat.com> wrote:

> On Fri, 2010-11-12 at 14:54 -0500, Simo Sorce wrote:
>
> > Adam why should security updates wait at all ?
> > Do you fear some packager will flag as security updates that are
> > not ? Surely we can deal with such maintainer if that happens...
>
> I don't have a hugely strong opinion either way, but the stated reason
> by those who do is that security updates can be broken just like any
> other. We don't have a magic 'infallible' switch on packagers which we
> toggle only when they're building a security update. :)

Oh sure I don't doubt that. But in this case we need to deal with the
lesser evil.
Is it more important to close a security bug with a (small) risk of
breaking a package ?
Or is it more important to (try to) test it and leave our users exposed
for a long time to a security threat ?

If we are not comfortable with treating all security issues the same we
can have a flag that skips testing only for "remote exploit" type of
security issues. That will reduce the number of exception to the most
dangerous cases.

What do you think ?

Simo.

--
Simo Sorce * Red Hat, Inc * New York
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel


All times are GMT. The time now is 06:50 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.