FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 11-14-2010, 01:03 PM
Bruno Wolff III
 
Default The new Update Acceptance Criteria are broken

On Sun, Nov 14, 2010 at 13:59:24 +0100,
Till Maas <opensource@till.name> wrote:
>
> If there are no security updates, people can not apply them. So what is
> worse? If people stop applying updates, then it is at least their
> decision. If there are no updates, people can only choose not to use

Many people are going to just pull updates. They aren't going to make a
decision on their own.

Security updates aren't all created equal. While the case that was
referenced in this was easily remotely exploitable, not all security
issues expose a system to that level of risk.

> The optimal case is to provide well tested security updates fast, but
> this is not what Fedora achieves. In my example there is no indication
> that the update was especially tested, because it did not get any karma.
> And it was not provided fast.

There is definitely a problem that needs fixing. But I don't think changing
the goal to untested security updates provided quickly is the preferred
solution.

Perhaps we should have a way to draw attention to high priority updates.
Generally we need more testers and need to make them more efficient.
(Test plans for packages can make testing more efficient and accurate.)
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-16-2010, 08:57 PM
Luke Macken
 
Default The new Update Acceptance Criteria are broken

On Mon, Nov 01, 2010 at 09:35:49AM -0600, Kevin Fenzi wrote:
> On Sun, 31 Oct 2010 10:16:41 -0400
> "Clyde E. Kunkel" <clydekunkel7734@cox.net> wrote:
>
> > On 10/31/2010 03:18 AM, Michael Schwendt wrote:
> > > Okay, feedback time.
> > >
> > > Lately, there have been several attempts at urging proventesters
> > > (and not just testers in general) to give positive karma for aging
> > > critpath updates. It also has been decided by someone (or maybe
> > > even a comittee) to spam proventesters daily with
> > > "[old_testing_critpath]" messages for all three dist releases, with
> > > no day to unsubscribe from that (other than leaving proventesters
> > > group, which is what at least one person has threatened with, or
> > > filtering those messages).
>
> Yeah, I am not sure at all how usefull those emails are.
> There are a variety of ways to see what things need testing, sending
> emails to proventesters a bunch isn't likely to be a very nice one.

I've disabled this behavior in git, and will update our production
instance this week. This spam is unnecessary now that this information
is available in the updates-testing digests sent to the test list.

luke
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-16-2010, 09:22 PM
Luke Macken
 
Default The new Update Acceptance Criteria are broken

On Fri, Nov 12, 2010 at 11:46:36AM -0800, Adam Williamson wrote:
> On Fri, 2010-11-12 at 14:32 -0500, Tom Lane wrote:
>
> > It's absolutely crystal clear to me that we don't have enough tester
> > manpower to make the current policy workable; it's past time to stop
> > denying that. I'd suggest narrowing the policy to a small number of
> > critical packages, for which there might be some hope of it actually
> > working as designed.
>
> BTW, another point worth noting is that there is not actually a specific
> rule against +1ing your own updates. It's 'frowned upon', I guess, but
> actually I think it's probably workable to say it's fine for the
> developer to +1 their own update in Bodhi: *as long as you actually have
> tested it*. For non-critpath updates especially. The Bodhi system is
> essentially an honor system anyway. So how I'd see this working is if it
> becomes clear that some maintainer is gaming the system by just +1ing
> everything they submit, even if it actually turns out to be broken, we
> look at saying they can't +1 their own updates any more. But if you
> actually are conscientiously testing your own updates, that's probably
> worth a +1 in Bodhi, for me.

The ability to +1 your own updates was disapproved by FESCo, and will be
disabled in a future version of bodhi.

https://fedorahosted.org/bodhi/ticket/277

luke
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-16-2010, 09:35 PM
François Cami
 
Default The new Update Acceptance Criteria are broken

On Fri, Nov 12, 2010 at 9:14 PM, Kevin Fenzi <kevin@scrye.com> wrote:
> On Fri, 12 Nov 2010 14:54:28 -0500
> Simo Sorce <ssorce@redhat.com> wrote:
>
>> Adam why should security updates wait at all ?
>> Do you fear some packager will flag as security updates that are not ?
>> Surely we can deal with such maintainer if that happens...
>
> No. The issue is that in the past sometimes security updates have been
> rushed out with no testing and broken things badly. ;(
>
> See http://fedoraproject.org/wiki/Updates_Lessons
> For some small number of examples (yes, anyone is welcome to please add
> others you have run into to the page).
>
> I know of at least dbus, bind, nss and a few others that were security
> updates and pushe out with no testing and turned out to break things.
>
> Perhaps security updates could have a smaller timeout?

This reminds me of the excellent "Timing the application of security
patches for optimal uptime" paper:
http://www.homeport.org/~adam/time-to-patch-usenix-lisa02.pdf
Maybe a smaller timeout would do, yet I don't think that we could use
a small enough (security-wise) timeout and still be safe from awful
regressions.

> Or a security group that tests them ?

People can't be expected to be knowledgeable or interested in all
packages that would need timely security testing.
In other words, telling all people in a group that an updated package
they don't care about is available would only add noise, while telling
relevant testers packages they care about are available may speed up
things.
For some reason, this reminds of RHN which sends emails only if
updates for the *installed* (relevant) packages are available.

We could create a subgroup of proventesters willing to enter
information about packages they know enough for them to test in a
timely manner. Or we could extend smolt to automate that task
(on-demand, of course). After all, we all tend to use what we install.

François
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-17-2010, 01:28 AM
Jon Masters
 
Default The new Update Acceptance Criteria are broken

On Tue, 2010-11-16 at 23:35 +0100, François Cami wrote:
> On Fri, Nov 12, 2010 at 9:14 PM, Kevin Fenzi <kevin@scrye.com> wrote:
> > On Fri, 12 Nov 2010 14:54:28 -0500
> > Simo Sorce <ssorce@redhat.com> wrote:
> >
> >> Adam why should security updates wait at all ?
> >> Do you fear some packager will flag as security updates that are not ?
> >> Surely we can deal with such maintainer if that happens...
> >
> > No. The issue is that in the past sometimes security updates have been
> > rushed out with no testing and broken things badly. ;(
> >
> > See http://fedoraproject.org/wiki/Updates_Lessons
> > For some small number of examples (yes, anyone is welcome to please add
> > others you have run into to the page).
> >
> > I know of at least dbus, bind, nss and a few others that were security
> > updates and pushe out with no testing and turned out to break things.
> >
> > Perhaps security updates could have a smaller timeout?
>
> This reminds me of the excellent "Timing the application of security
> patches for optimal uptime" paper:
> http://www.homeport.org/~adam/time-to-patch-usenix-lisa02.pdf
> Maybe a smaller timeout would do, yet I don't think that we could use
> a small enough (security-wise) timeout and still be safe from awful
> regressions.

Funnily enough, that paper concludes that 10 days is an optimal time to
delay patch application in order to avoid getting a bad patch that will
need later revision. This is close to the 7 days used for pre-release
updates in the Updates Policy. So I would take this as another useful
data point demonstrating that one doesn't need to push security updates
instantaneously to stable, but that they can wait for testing results.

(no policy is immune from huge day zero vulnerabilities with massive
exploits, but I'm certain that if there were such an incident with this
policy, an exception could be made if the impact was significant)

Jon.


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-17-2010, 05:10 AM
Chris Wright
 
Default The new Update Acceptance Criteria are broken

* Jon Masters (jonathan@jonmasters.org) wrote:
> On Tue, 2010-11-16 at 23:35 +0100, François Cami wrote:
> > On Fri, Nov 12, 2010 at 9:14 PM, Kevin Fenzi <kevin@scrye.com> wrote:
> > > On Fri, 12 Nov 2010 14:54:28 -0500
> > > Simo Sorce <ssorce@redhat.com> wrote:
> > >
> > >> Adam why should security updates wait at all ?
> > >> Do you fear some packager will flag as security updates that are not ?
> > >> Surely we can deal with such maintainer if that happens...
> > >
> > > No. The issue is that in the past sometimes security updates have been
> > > rushed out with no testing and broken things badly. ;(
> > >
> > > See http://fedoraproject.org/wiki/Updates_Lessons
> > > For some small number of examples (yes, anyone is welcome to please add
> > > others you have run into to the page).
> > >
> > > I know of at least dbus, bind, nss and a few others that were security
> > > updates and pushe out with no testing and turned out to break things.
> > >
> > > Perhaps security updates could have a smaller timeout?
> >
> > This reminds me of the excellent "Timing the application of security
> > patches for optimal uptime" paper:
> > http://www.homeport.org/~adam/time-to-patch-usenix-lisa02.pdf
> > Maybe a smaller timeout would do, yet I don't think that we could use
> > a small enough (security-wise) timeout and still be safe from awful
> > regressions.
>
> Funnily enough, that paper concludes that 10 days is an optimal time to
> delay patch application in order to avoid getting a bad patch that will
> need later revision. This is close to the 7 days used for pre-release
> updates in the Updates Policy. So I would take this as another useful
> data point demonstrating that one doesn't need to push security updates
> instantaneously to stable, but that they can wait for testing results.

I'd not place as much emphasis in the exact number as the concept (10
days was the conclusion based on the CVE data available at the time,
but there's a lot more data now). Security fixes are not magically
immune from testing and validation.

> (no policy is immune from huge day zero vulnerabilities with massive
> exploits, but I'm certain that if there were such an incident with this
> policy, an exception could be made if the impact was significant)

Yup, I agree. Basic risk assessment.

thanks,
-chris
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-18-2010, 05:09 PM
Till Maas
 
Default The new Update Acceptance Criteria are broken

On Sun, Nov 14, 2010 at 08:03:35AM -0600, Bruno Wolff III wrote:
> On Sun, Nov 14, 2010 at 13:59:24 +0100,
> Till Maas <opensource@till.name> wrote:

> > The optimal case is to provide well tested security updates fast, but
> > this is not what Fedora achieves. In my example there is no indication
> > that the update was especially tested, because it did not get any karma.
> > And it was not provided fast.
>
> There is definitely a problem that needs fixing. But I don't think changing
> the goal to untested security updates provided quickly is the preferred
> solution.

The root cause for the new update acceptance criteria was that there
where updates that broke systems. Now with the criteria enforced,
systems are broken worse according to the collection of bad update
examples, because of updates not being pushed to stable. This was
something that was always being highlighted as a potential problem iirc
and yet it happend.

> Perhaps we should have a way to draw attention to high priority updates.
> Generally we need more testers and need to make them more efficient.
> (Test plans for packages can make testing more efficient and accurate.)

Yes, more manpower would help, but where should it come from?

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-20-2010, 05:04 AM
Adam Williamson
 
Default The new Update Acceptance Criteria are broken

On Tue, 2010-11-16 at 17:22 -0500, Luke Macken wrote:
> On Fri, Nov 12, 2010 at 11:46:36AM -0800, Adam Williamson wrote:
> > On Fri, 2010-11-12 at 14:32 -0500, Tom Lane wrote:
> >
> > > It's absolutely crystal clear to me that we don't have enough tester
> > > manpower to make the current policy workable; it's past time to stop
> > > denying that. I'd suggest narrowing the policy to a small number of
> > > critical packages, for which there might be some hope of it actually
> > > working as designed.
> >
> > BTW, another point worth noting is that there is not actually a specific
> > rule against +1ing your own updates. It's 'frowned upon', I guess, but
> > actually I think it's probably workable to say it's fine for the
> > developer to +1 their own update in Bodhi: *as long as you actually have
> > tested it*. For non-critpath updates especially. The Bodhi system is
> > essentially an honor system anyway. So how I'd see this working is if it
> > becomes clear that some maintainer is gaming the system by just +1ing
> > everything they submit, even if it actually turns out to be broken, we
> > look at saying they can't +1 their own updates any more. But if you
> > actually are conscientiously testing your own updates, that's probably
> > worth a +1 in Bodhi, for me.
>
> The ability to +1 your own updates was disapproved by FESCo, and will be
> disabled in a future version of bodhi.
>
> https://fedorahosted.org/bodhi/ticket/277

hum, that wasn't well publicised, and I wasn't aware of it. (I should
probably show up to more FESCo meetings...picture FESCo members going
'no, no, really, it's fine!') I'd disagree, for the reasons above.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-20-2010, 08:49 PM
Kevin Fenzi
 
Default The new Update Acceptance Criteria are broken

On Fri, 19 Nov 2010 22:04:24 -0800
Adam Williamson <awilliam@redhat.com> wrote:

...snip...

> > https://fedorahosted.org/bodhi/ticket/277
>
> hum, that wasn't well publicised, and I wasn't aware of it. (I should
> probably show up to more FESCo meetings...picture FESCo members going
> 'no, no, really, it's fine!') I'd disagree, for the reasons above.

Well, my thought on it is:

* As maintainer, shouldn't you be testing your updates already? Granted
there's often no way you could test everything, but at least
installing it and confirming the bug(s) you claim are fixed are
fixed?

* Having just one pair of eyes (even if they are exceptionally talented
ones) can lead to things slipping through. I know personally I have
messed up and tested something, confirmed the bug was fixed, then due
to one of: trying to clean up the commit before building without
another test cycle, testing on a machine with different package
versions, etc the update I push out is not the one that really fixes
the issues or introduces some other issue. Unless you have a really
really good test workflow this kind of thing can happen.

We can surely revisit this if folks like... but if we do, we might
consider for non critpath packages just re-enabling a 'push to stable',
as thats what it would allow with less clicking.

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-20-2010, 09:09 PM
Adam Williamson
 
Default The new Update Acceptance Criteria are broken

On Sat, 2010-11-20 at 14:49 -0700, Kevin Fenzi wrote:
> On Fri, 19 Nov 2010 22:04:24 -0800
> Adam Williamson <awilliam@redhat.com> wrote:
>
> ...snip...
>
> > > https://fedorahosted.org/bodhi/ticket/277
> >
> > hum, that wasn't well publicised, and I wasn't aware of it. (I should
> > probably show up to more FESCo meetings...picture FESCo members going
> > 'no, no, really, it's fine!') I'd disagree, for the reasons above.
>
> Well, my thought on it is:
>
> * As maintainer, shouldn't you be testing your updates already? Granted
> there's often no way you could test everything, but at least
> installing it and confirming the bug(s) you claim are fixed are
> fixed?

I do. I don't believe all maintainers do. It's pretty hard to explain
why updates that completely prevent the app in question from working, or
even prevent the system from booting, got pushed in the past, if all
maintainers actually test their updates.

The advantage of doing it my way (allowing maintainers to test their own
updates and file bodhi feedback, but requiring bodhi feedback) is that
it leaves an audit trail: it requires the maintainer to effectively make
an explicit public declaration that they tested the update and it
worked, rather than just relying on the implied 'oh of course they must
have tested it'. What this means is that if we come across cases where a
maintainer builds an update, submits it, files bodhi feedback saying
they've tested it, and it turns out to be completely broken in a way
they should have caught if they tested it, we now have all the necessary
evidence to take some kind of sanctions against that packager.

Of course, the idea would be that we'd never have to do that, because
the fact that the above is the case would be sufficient motivation to
ensure that packagers really *do* test their updates properly.

> * Having just one pair of eyes (even if they are exceptionally talented
> ones) can lead to things slipping through. I know personally I have
> messed up and tested something, confirmed the bug was fixed, then due
> to one of: trying to clean up the commit before building without
> another test cycle, testing on a machine with different package
> versions, etc the update I push out is not the one that really fixes
> the issues or introduces some other issue. Unless you have a really
> really good test workflow this kind of thing can happen.
>
> We can surely revisit this if folks like... but if we do, we might
> consider for non critpath packages just re-enabling a 'push to stable',
> as thats what it would allow with less clicking.
>

See above for what I consider the advantage of preserving the karma
requirement but allowing the maintainer to provide it. This is for
non-critpath packages, of course. I don't think this should be allowed
for critpath.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 06:39 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org