FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 11-20-2010, 09:35 PM
Kevin Fenzi
 
Default Updates Criteria Summary/Brainstorming

ok, I dug through the devel list for the last month or two and wrote
down all the various ideas folks have come up with to change/improve
things.

Here (in no particular order) are the ideas and some notes from me on
how we could enable them. Please feel free to add new (actual/concrete
ideas or notes):

* Just drop all the requirements/go back to before we had any updates
criteria.

* Change FN-1 to just security and major bugfix

This may be hard to enforce or figure out if something is a major bugfix.

* allow packages with a %check section to go direct to stable

Bodhi would have to have a list or some way to note these packages,
it would also need to change as they were added/removed. Perhaps it could
just be an AutoQA +1 for having a check section?
On the con side, some checks may be simple and may not note things
that are fedora only issues.

* setup a remote test env that people could use to test things.

This would be lovely with some kind of cloud setup. Have a set of base kickstart
files and a package/tester could request an instance, install the update, test, and then
the instance would be removed. We would need a timelimit of some kind,
and not sure there is any HW in infrastructure for this, but it would sure be cool.
Perhaps working with the cloud sig we could use EC2 instances for this?

* require testing only for packages where people have signed up to be testers

Packages without 'official' testers could bypass testing or have some lower karma
requirement. We would need for this a list of packages that have had people sign
up to test.

* Ask maintainers to provide test cases / test cases in wiki for each package?

Test cases are not easy to make, many maintainers won't can't do so, but
it would be lovely to have even a base checklist of things that should work
in the package everytime.

* have a way to get interested testers notified on bodhi updates for packages
they care about.

We would need to add some kind of tester list to pkgdb, and bodhi would need to be
able to get this to mail them when a update changed state.
We may not get many people signing up for some packages, but this might be a
good way to know what packages we have testers for and get them more involved
in testing. Ideally it could mail them on update submission at least.

* reduced karma requirement on other releases when one has gone stable

Bodhi would need to note when a update went to stable if the exact same
version (with dist tag differences) was in testing for other releases.
It could then allow less karma to go stable, or add +1 from the other
update going stable.

Other concrete ideas?

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-20-2010, 10:47 PM
François Cami
 
Default Updates Criteria Summary/Brainstorming

On Sat, Nov 20, 2010 at 11:35 PM, Kevin Fenzi <kevin@scrye.com> wrote:
> ok, I dug through the devel list for the last month or two and wrote
> down all the various ideas folks have come up with to change/improve
> things.
>
> Here (in no particular order) are the ideas and some notes from me on
> how we could enable them. Please feel free to add new (actual/concrete
> ideas or notes):
>
> * Just drop all the requirements/go back to before we had any updates
> *criteria.

Before we go that route, maybe we should understand why we decided on
the updates criteria. If I remember correctly, this was decided
because some updates introduced regressions in stable Fedora releases,
upsetting users. Do we really want to go back?

> * Change FN-1 to just security and major bugfix
>
> This may be hard to enforce or figure out if something is a major bugfix.

My vote would be security-only, because people desiring the latest
upstream code apparently upgrade to the latest Fedora release as soon
as it's baked (and sometimes earlier), therefore there is no need to
upgrade the packages in n-1.
And the main reason why a user doesn't upgrade his box seems rooted in
the wish to avoid unnecessary downtime or extra work - after all, they
can skip a release since n-1 is supported. So these users want
*stability*, not anything disruptive, and it's OK if the software they
use is a bit older than what is available in the latest Fedora. The
latest software is only an upgrade away after all.

Any major non-security bugfix needed in n-1 wasn't detected for 6
months or so. How "major" is that?
Or maybe it was introduced by an update and not caught early enough
because we couldn't test the update.

> * allow packages with a %check section to go direct to stable
>
> Bodhi would have to have a list or some way to note these packages,
> it would also need to change as they were added/removed. Perhaps it could
> just be an AutoQA +1 for having a check section?
> On the con side, some checks may be simple and may not note things
> that are fedora only issues.

More test automation would be an excellent way to validate updates.
It's not applicable everywhere, but it should help a lot.

> * require testing only for packages where people have signed up to be testers
>
> Packages without 'official' testers could bypass testing or have some lower karma
> requirement. We would need for this a list of packages that have had people sign
> up to test.

I'm not too keen on that one, although I don't see a way out. I don't
have a big enough network at home to extensively test Nagios or
389-DS, for instance. Sanity tests of these are possible, but I won't
easily (with my current resources) catch performance regressions or
some kind of instability under load.

> * Ask maintainers to provide test cases / test cases in wiki for each package?
>
> Test cases are not easy to make, many maintainers won't can't do so, but
> it would be lovely to have even a base checklist of things that should work
> in the package everytime.

Especially since test cases could form the basis of automated tests
later on. And I'm pretty sure any time spent doing test automation or
writing test cases would be worth it down the road, as we'd have more
confidence in subsequent updates.

> * have a way to get interested testers notified on bodhi updates for packages
> they care about.
>
> We would need to add some kind of tester list to pkgdb, and bodhi would need to be
> able to get this to mail them when a update changed state.
> We may not get many people signing up for some packages, but this might be a
> good way to know what packages we have testers for and get them more involved
> in testing. Ideally it could mail them on update submission at least.
>
> * reduced karma requirement on other releases when one has gone stable
>
> Bodhi would need to note when a update went to stable if the exact same
> version (with dist tag differences) was in testing for other releases.
> It could then allow less karma to go stable, or add +1 from the other
> update going stable.

This sounds good on paper, yet we'll have to remember it cannot
eliminate testing altogether due to packages using shared libraries
that might not be of the same version across releases. Still, given
we're short-handed, that's a good solution.

> Other concrete ideas?

These are all technical ideas. We need to know what experience we want
to provide users with in both Fedora n and n-1 before deciding which
technical idea(s) to implement.

My own vision is that:
* updates should not be less tested than what's in an actual release -
and the QA/RE teams do an amazing amount of testing prior to releases.
I would love to see more maintainers at the Go/NoGo meetings for
instance.
* updates should not disrupt user experience. Fedora, being
community-supported, is probably unsuitable for many business tasks.
Let's not invalidate Fedora for less technically minded people.
* Fedora releases do have bugs. After any release, we should strive to
lower the total bug count, not introducing massive changes that will
in most cases introduce more defects.
* upgrading at least some software in Fedora n is OK, yet upgrading
Fedora n-1 packages shouldn't be our priority. Most of our users who
want the latest software should (and probably do) run the latest
Fedora anyway.

I encourage you all to read Máirín's excellent post on new users and
free software.
http://mairin.wordpress.com/2010/10/01/you-must-be-this-tall-to-ride-__/
We need to ask ourselves what happens to a user when the software
he/she relies on everyday ceases to work after an update, how
difficult it is for these people to find answers, and a fix, online.

Please note that I am *not* saying all updates are evil. Yet I think
that avoiding massive breakage in critpath packages should be one of
our priorities.

François
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 04:00 AM
Kevin Kofler
 
Default Updates Criteria Summary/Brainstorming

Kevin Fenzi wrote:
> * Just drop all the requirements/go back to before we had any updates
> criteria.

That's really the only way to go. The policy failed, it's time to withdraw
it. All the other proposed solutions require even more complexity in the
software and policies, for little to no gain.


> * Change FN-1 to just security and major bugfix
>
> This may be hard to enforce or figure out if something is a major bugfix.

Indeed, this obviously doesn't work.

> * allow packages with a %check section to go direct to stable

%check
echo 'Success!'

Is that OK? If not, what is? How much testing do you want to require?

And most importantly, it doesn't solve the problems for those many packages
for which automated testing is not feasible and/or not useful.

> * setup a remote test env that people could use to test things.

That doesn't solve the time issue, only the "I don't have a Fedora n
environment" issue, and not everything can be tested properly in such a
setup. (Hardware-specific issues, latency-critical software etc.)

> * require testing only for packages where people have signed up to be
> testers

The maintainers know best whether their packages have sufficient testers or
not, just let them decide on how much feedback to wait for before going
stable! A boolean is not sufficient to accurately describe the situation,
e.g. requiring a karma of 5 may make sense for something like the kernel,
but not for a package with only 4 testers in total, and also the testers
available for a given Fedora release matter (a number which changes over
time, and you can't really rely on testers updating their data each time
they upgrade their system(s)).

> * Ask maintainers to provide test cases / test cases in wiki for each
> package?

There are many packages where that's just not feasible. (Good luck trying to
provide an exhaustive set of test cases for e.g. kdebase-workspace!) It's
also a lot of extra work for the maintainers.

> * have a way to get interested testers notified on bodhi updates for
> packages they care about.

That doesn't solve the problem of there not being interested testers in the
first place.

> * reduced karma requirement on other releases when one has gone stable

In principle, that makes sense. It might solve part of the issues if the
"reduced" karma requirement is zero. (Otherwise it's just useless, since we
can already set it to 1.) But you'd have to allow the maintainer to tell
Bodhi what 2 updates are the same. "Same EVR minus disttags" as you propose
has both false positives and false negatives. And why not avoid all this
complexity by just always letting the maintainer decide? They know best how
much value to attribute to feedback from identical or similar updates for
other releases in the specific case at hand.


In short, my proposal:
1. discontinue the current update acceptance enforcement (in particular,
reenable direct stable pushes, and of course allow pushes from testing to
stable at any moment at the maintainer's discretion),
2. drop the aggregated numeric karma score, which is devoid of any actual
significance, and the autokarma (mis)feature that goes with it (keep only
the +1/0/-1 emoticons on the individual comments),
3. write some recommendations which should GUIDE the maintainers on how to
handle updates, but NOT FORCE anything on them (experienced maintainers
follow many unwritten rules, writing them down can certainly help guiding
less experienced maintainers towards doing the right thing),
4. TRUST maintainers to make the right decisions on when an update is stable
enough to be pushed, also considering the impact of NOT pushing the update
immediately (which has worked very well, despite some claims to the contrary
based on isolated incidents blown way out of proportion).

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 09:00 AM
Till Maas
 
Default Updates Criteria Summary/Brainstorming

On Sat, Nov 20, 2010 at 03:35:43PM -0700, Kevin Fenzi wrote:

> * require testing only for packages where people have signed up to be testers
>
> Packages without 'official' testers could bypass testing or have some lower karma
> requirement. We would need for this a list of packages that have had people sign
> up to test.

I guess this can be somehow automated. E.g. change Bodhi to drop the
karma requirements for packages that had e.g. two subsequent updates
without any Bodhi feedback and re-enable it if they get feedback.

> * Ask maintainers to provide test cases / test cases in wiki for each package?
>
> Test cases are not easy to make, many maintainers won't can't do so, but
> it would be lovely to have even a base checklist of things that should work
> in the package everytime.

Afaik, most of the packages won't be tested and therefore the test cases
won't be read. So this should be only a requirement if there are testers
for a certain package.

> * reduced karma requirement on other releases when one has gone stable
>
> Bodhi would need to note when a update went to stable if the exact same
> version (with dist tag differences) was in testing for other releases.
> It could then allow less karma to go stable, or add +1 from the other
> update going stable.
>
> Other concrete ideas?

All of this could be combined. E.g. packages with enough testers get
test cases and need to fulfill stronger criteria. Packages with not so
many testers get test cases and only need to fulfil that similar
updates need to receive good karma on one Fedora release.

Off course more automated testing is missing instead of all the manual
requirements.

Another idea would be to add a flag in bodhi to track whether a certain
testing update has been installed and then have a tool to report this
automatically back to Bodhi. Then there would be some information about
silent testers, which can be used as a criterion, too.

Also it could be made easier for maintainers to identify problematic
updates, e.g. by warning that the dependencies or provides of an update
changed when the update is created.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 09:29 AM
Kevin Kofler
 
Default Updates Criteria Summary/Brainstorming

Till Maas wrote:
> All of this could be combined. E.g. packages with enough testers get
> test cases and need to fulfill stronger criteria. Packages with not so
> many testers get test cases and only need to fulfil that similar
> updates need to receive good karma on one Fedora release.
>
> Off course more automated testing is missing instead of all the manual
> requirements.
>
> Another idea would be to add a flag in bodhi to track whether a certain
> testing update has been installed and then have a tool to report this
> automatically back to Bodhi. Then there would be some information about
> silent testers, which can be used as a criterion, too.

All this stuff would really complicate Bodhi a lot, and make it even more
bug-prone than it already is.

Why do we need to code this in software? Why can't we just make use of the
wonderful deduction machine called a "brain", which every maintainer SHOULD
have?

> Also it could be made easier for maintainers to identify problematic
> updates, e.g. by warning that the dependencies or provides of an update
> changed when the update is created.

That's something useful, and in fact AutoQA is supposed to do that (and
we're still waiting for that!).

Doing repetitive consistency checks and warning the maintainer about
potential or definite problems is what software is good at. Taking decisions
is not!

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 10:42 AM
Michael Schwendt
 
Default Updates Criteria Summary/Brainstorming

On Sat, 20 Nov 2010 15:35:43 -0700, Kevin wrote:

> Other concrete ideas?

As a beginning, let's limit this thread to at most one message per person
per day.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 06:00 PM
Toshio Kuratomi
 
Default Updates Criteria Summary/Brainstorming

On Sat, Nov 20, 2010 at 03:35:43PM -0700, Kevin Fenzi wrote:
> ok, I dug through the devel list for the last month or two and wrote
> down all the various ideas folks have come up with to change/improve
> things.
>
> Here (in no particular order) are the ideas and some notes from me on
> how we could enable them. Please feel free to add new (actual/concrete
> ideas or notes):
>
> * Just drop all the requirements/go back to before we had any updates
> criteria.
>
> * Change FN-1 to just security and major bugfix
>
> This may be hard to enforce or figure out if something is a major bugfix.
>
> * allow packages with a %check section to go direct to stable
>
> Bodhi would have to have a list or some way to note these packages,
> it would also need to change as they were added/removed. Perhaps it could
> just be an AutoQA +1 for having a check section?
> On the con side, some checks may be simple and may not note things
> that are fedora only issues.
>
> * setup a remote test env that people could use to test things.
>
> This would be lovely with some kind of cloud setup. Have a set of base kickstart
> files and a package/tester could request an instance, install the update, test, and then
> the instance would be removed. We would need a timelimit of some kind,
> and not sure there is any HW in infrastructure for this, but it would sure be cool.
> Perhaps working with the cloud sig we could use EC2 instances for this?
>
> * require testing only for packages where people have signed up to be testers
>
> Packages without 'official' testers could bypass testing or have some lower karma
> requirement. We would need for this a list of packages that have had people sign
> up to test.
>
> * Ask maintainers to provide test cases / test cases in wiki for each package?
>
> Test cases are not easy to make, many maintainers won't can't do so, but
> it would be lovely to have even a base checklist of things that should work
> in the package everytime.
>
> * have a way to get interested testers notified on bodhi updates for packages
> they care about.
>
> We would need to add some kind of tester list to pkgdb, and bodhi would need to be
> able to get this to mail them when a update changed state.
> We may not get many people signing up for some packages, but this might be a
> good way to know what packages we have testers for and get them more involved
> in testing. Ideally it could mail them on update submission at least.
>
> * reduced karma requirement on other releases when one has gone stable
>
> Bodhi would need to note when a update went to stable if the exact same
> version (with dist tag differences) was in testing for other releases.
> It could then allow less karma to go stable, or add +1 from the other
> update going stable.
>
> Other concrete ideas?
>

Security update ideas:

* Security updates may push directly to stable -- this would depend on
our weighing of costs: a security update may be broken. OTOH, if we
don't get security updates out fast, is that a worse danger to our users?
I think I'd like to try one of the other options below first but those
require a bit more work/coordination so we need to think of the cost to
implement as well.

* Ask the QA team to commit to testing security updates. The idea is that
updates flagged security have a dedicated pool of testers that will check
them and get them out ASAP.

* Security updates have a small timeout period -- there's two questions
here: Is a week (as is currently the case for non-critpath) too long?
Can wehave a timeout even for critpath packages so a security update
doesn't get held up forever?

Critpath ideas:

* critpath package timeout -- let's have a timeout period even for critpath
packages. Perhaps this could be longer than for non-critpath. The big
issue that people have observed with depending on timeouts, though, is
that pushing new updates in the meantime resets the time that a package
needs to wait to get to stable. Could bodhi be changed to let multiple
packages be in the testing repository at one time and only obsolete them
when a newer package enters the stable repo? That would allow longer
timeout periods to make more sense.

Lack of manpower ideas:

* Allow anonymous karma to count. Anonymous karma would allow more people
who report bugs in bugzilla to add karma in bodhi without having to get
a second account in the Fedora Account System. For critpath packages,
we're already mandating that a proventester must give karma so we're
making sure that someone with an account is giving feedback there.

* Do we believe that there's value in silent testing (knowing that people
have a package installed from testing but have not submitted karma either
way?) If so, create a tool that ties into yumdb and if the user opts in,
submits that data to us. Then add karma based upon that data.

Variation on an option you recorded:
* Drop testing requirement until AutoQA can give karma. Rework based on
what tests auto qacan perform.

-Toshio
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 06:14 PM
Andre Robatino
 
Default Updates Criteria Summary/Brainstorming

Toshio Kuratomi <a.badger <at> gmail.com> writes:

> Lack of manpower ideas:
>
> * Allow anonymous karma to count. Anonymous karma would allow more people
> who report bugs in bugzilla to add karma in bodhi without having to get
> a second account in the Fedora Account System. For critpath packages,
> we're already mandating that a proventester must give karma so we're
> making sure that someone with an account is giving feedback there.

My feeling is that it would be better for Bodhi to always require a login. Even
Bugzilla does that. I suspect that a lot of people who give anonymous karma
don't realize that it doesn't count, and would have created an account if they
did. And using an account allows people to build a reputation, which is useful
in case Bodhi is ever besieged by malicious comments and is forced to disable
some accounts, or disable anonymous comments completely.




--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 08:49 PM
Kevin Kofler
 
Default Updates Criteria Summary/Brainstorming

Toshio Kuratomi wrote:
> packages. Perhaps this could be longer than for non-critpath. The big
> issue that people have observed with depending on timeouts, though, is
> that pushing new updates in the meantime resets the time that a package
> needs to wait to get to stable. Could bodhi be changed to let multiple
> packages be in the testing repository at one time and only obsolete them
> when a newer package enters the stable repo? That would allow longer
> timeout periods to make more sense.

The problem with that is that only one of the updates is going to be
actually tested at a time.

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 11-21-2010, 08:58 PM
Jesse Keating
 
Default Updates Criteria Summary/Brainstorming

On 11/21/10 11:00 AM, Toshio Kuratomi wrote:
> meantime resets the time that a package
> needs to wait to get to stable. Could bodhi be changed to let multiple
> packages be in the testing repository at one time and only obsolete them
> when a newer package enters the stable repo? That would allow longer
> timeout periods to make more sense.

It would be difficult client side to select which set of packages to
update to, generally people will just get the latest set, not the
earlier set.

--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 08:03 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org