FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 11-12-2010, 07:14 PM
Kevin Fenzi
 
Default The new Update Acceptance Criteria are broken

On Fri, 12 Nov 2010 14:54:28 -0500
Simo Sorce <ssorce@redhat.com> wrote:

> Adam why should security updates wait at all ?
> Do you fear some packager will flag as security updates that are not ?
> Surely we can deal with such maintainer if that happens...

No. The issue is that in the past sometimes security updates have been
rushed out with no testing and broken things badly. ;(

See http://fedoraproject.org/wiki/Updates_Lessons
For some small number of examples (yes, anyone is welcome to please add
others you have run into to the page).

I know of at least dbus, bind, nss and a few others that were security
updates and pushe out with no testing and turned out to break things.

Perhaps security updates could have a smaller timeout?
Or a security group that tests them ?

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-12-2010, 11:09 PM
"Clyde E. Kunkel"
 
Default The new Update Acceptance Criteria are broken

On 11/12/2010 02:32 PM, Tom Lane wrote:
> Till Maas<opensource@till.name> writes:
>> <snip>
>
> It's absolutely crystal clear to me that we don't have enough tester
> manpower to make the current policy workable; it's past time to stop
> denying that. I'd suggest narrowing the policy to a small number of
> critical packages, for which there might be some hope of it actually
> working as designed.
>
> regards, tom lane


Test cases would help alleviate manpower issues. Many of the security
updates and regular updates are outside my area and I feel some
frustration that I have to bypass providing karma; however, I am used to
doing QA work with test cases. Are they so hard to provide? Maybe
certain updates should have test cases, i.e., security updates and
critical path updates.

Regards,
OldFart
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 03:14 AM
Tom Lane
 
Default The new Update Acceptance Criteria are broken

"Clyde E. Kunkel" <clydekunkel7734@cox.net> writes:
> On 11/12/2010 02:32 PM, Tom Lane wrote:
>> It's absolutely crystal clear to me that we don't have enough tester
>> manpower to make the current policy workable; it's past time to stop
>> denying that. I'd suggest narrowing the policy to a small number of
>> critical packages, for which there might be some hope of it actually
>> working as designed.

> Test cases would help alleviate manpower issues. Many of the security
> updates and regular updates are outside my area and I feel some
> frustration that I have to bypass providing karma; however, I am used to
> doing QA work with test cases. Are they so hard to provide? Maybe
> certain updates should have test cases, i.e., security updates and
> critical path updates.

The major packages that I work with have regression test suites,
which in fact get run as part of the RPM build sequence. It's not
apparent to me that I should need to invent some more tests.

The likely failure cases that I can see are of two types:

1. Upstream screwed up and introduced a regression into what was
supposed to be a minor bug-fix or security update. This does happen,
for sure, but there's pretty much 0 chance that I as packager am going
to catch it if it gets past the built-in regression tests.
Unfortunately, there is also pretty much 0 chance that Fedora testers
are going to notice such a problem in the limited time window for sanity
testing. It hasn't ever happened for any of my packages that Fedora
testers caught such things in time.

2. I screwed up and introduced a packaging bug, for instance bad
dependencies or inability to "yum update". That's been known to happen
too. But I have a lot more faith in autoqa being able to catch that
kind of problem in a timely fashion than I do in manual testing catching
it.

I guess what this boils down to is that I'd be happier with the testing
process if it were actually successful at finding problems. In my
experience, it's a week's delay for exactly zero return.

regards, tom lane
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 03:43 AM
Adam Williamson
 
Default The new Update Acceptance Criteria are broken

On Fri, 2010-11-12 at 23:14 -0500, Tom Lane wrote:

> 2. I screwed up and introduced a packaging bug, for instance bad
> dependencies or inability to "yum update". That's been known to happen
> too. But I have a lot more faith in autoqa being able to catch that
> kind of problem in a timely fashion than I do in manual testing catching
> it.

In the long run so do we, but right now, autoqa is not hooked up to the
build process in any way. It's manual testing or nothing.

> I guess what this boils down to is that I'd be happier with the testing
> process if it were actually successful at finding problems. In my
> experience, it's a week's delay for exactly zero return.

It does find problems. Though, by what you say, not in your packages, so
I know where you're coming from; but we've certainly caught a positive
integer amount of bugs with the process.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 08:21 AM
Till Maas
 
Default The new Update Acceptance Criteria are broken

On Fri, Nov 12, 2010 at 01:14:12PM -0700, Kevin Fenzi wrote:

> No. The issue is that in the past sometimes security updates have been
> rushed out with no testing and broken things badly. ;(
>
> See http://fedoraproject.org/wiki/Updates_Lessons
> For some small number of examples (yes, anyone is welcome to please add
> others you have run into to the page).

The documented issues do not seem to be as bad as a system being
exploited. It is only about dependency breakage or services not working
anymore. There is no major data corruption requiring access to backups
and restoring the whole system. But this is what people using Fedora
with proftpd and being exploited have to do.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 08:24 AM
Till Maas
 
Default The new Update Acceptance Criteria are broken

On Fri, Nov 12, 2010 at 11:19:22AM -0800, Adam Williamson wrote:

> Thanks for flagging this up.
>
> I'm wondering if perhaps we should devise a system - maybe a sub-group
> of proventesters - to ensure timely testing of security updates. wdyt?

I am not sure if a smaller group would help here. But what is certainly
missing is proper monitoring of updates that need to be tested asap and
notify testers or people in charge of untested updates.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 01:22 PM
Matthew Garrett
 
Default The new Update Acceptance Criteria are broken

On Sat, Nov 13, 2010 at 10:21:30AM +0100, Till Maas wrote:

> The documented issues do not seem to be as bad as a system being
> exploited. It is only about dependency breakage or services not working
> anymore. There is no major data corruption requiring access to backups
> and restoring the whole system. But this is what people using Fedora
> with proftpd and being exploited have to do.

If security updates break functionality then people will stop applying
security updates.

--
Matthew Garrett | mjg59@srcf.ucam.org
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-13-2010, 02:44 PM
"Clyde E. Kunkel"
 
Default The new Update Acceptance Criteria are broken

On 11/12/2010 11:14 PM, Tom Lane wrote:
> "Clyde E. Kunkel"<clydekunkel7734@cox.net> writes:
>> <snip>

> The major packages that I work with have regression test suites,
> which in fact get run as part of the RPM build sequence. It's not
> apparent to me that I should need to invent some more tests.
>

I did not know that. Good to know. Would it help if the test cases
were mentioned so their use could be considered in providing karma? Or,
even if they were made available?

Regards,

OldFart
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-14-2010, 04:52 AM
Matt McCutchen
 
Default The new Update Acceptance Criteria are broken

On Sat, 2010-11-13 at 14:22 +0000, Matthew Garrett wrote:
> On Sat, Nov 13, 2010 at 10:21:30AM +0100, Till Maas wrote:
>
> > The documented issues do not seem to be as bad as a system being
> > exploited. It is only about dependency breakage or services not working
> > anymore. There is no major data corruption requiring access to backups
> > and restoring the whole system. But this is what people using Fedora
> > with proftpd and being exploited have to do.
>
> If security updates break functionality then people will stop applying
> security updates.

That may be true in general, but I think Till has given a compelling
example in which many (most?) users would prefer an update with some
probability of being broken to no update. If necessary, we could have a
separate repository of "urgent" updates that sysadmins could choose to
enable or not based on their security and stability needs.

--
Matt

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-14-2010, 11:59 AM
Till Maas
 
Default The new Update Acceptance Criteria are broken

On Sat, Nov 13, 2010 at 02:22:42PM +0000, Matthew Garrett wrote:
> On Sat, Nov 13, 2010 at 10:21:30AM +0100, Till Maas wrote:
>
> > The documented issues do not seem to be as bad as a system being
> > exploited. It is only about dependency breakage or services not working
> > anymore. There is no major data corruption requiring access to backups
> > and restoring the whole system. But this is what people using Fedora
> > with proftpd and being exploited have to do.
>
> If security updates break functionality then people will stop applying
> security updates.

If there are no security updates, people can not apply them. So what is
worse? If people stop applying updates, then it is at least their
decision. If there are no updates, people can only choose not to use
Fedora. E.g. either build the applications themselves or use another
distribution. But this is not a viable goal.

The optimal case is to provide well tested security updates fast, but
this is not what Fedora achieves. In my example there is no indication
that the update was especially tested, because it did not get any karma.
And it was not provided fast.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 09:36 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org