FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 04-18-2011, 12:55 PM
Kamil Paral
 
Default AutoQA: distro congestion?

I've created a short blogpost about this issue:

http://kparal.wordpress.com/2011/04/18/autoqa-upgradepath-vs-updates-to-multiple-fedora-releases/
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 08:30 AM
Axel Thimm
 
Default AutoQA: distro congestion?

On Sun, 2011-04-17 at 08:01 +0200, Kevin Kofler wrote:
> Axel Thimm wrote:
> > Are you sure? I requested a push of the packages to stable (some were in
> > testing for a week, others were security updates) and the message was
> > that it doesn't pass AutoQA, so it converted to request to push only to
> > testing and indeed bodhi has marked the request as to testing only.
>
> It did that because you changed it.
>
> If you had kept the request to stable, it would have been pushed to stable.
>
> AutoQA is still in testing phase, there is no enforcement yet.

Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
prominently displays:

Bodhi is now enforcing the Package Update Acceptance
Criteria across all Fedora releases.

And the conversion to testing was mentioned when I tried to push the
package, not later, when a human could process the request.

Currently the autoqa process is very muddy to me and probably other
packagers as well:

* There are contradictory statements on whether autoqa will be
enforced or informative only. bodhi claims it is in place, Tim
claims it will be in place by F16 and other claim it will always
be informative only. For me it looks like bodhi rejects my
packages based on autoqa partly without any autoqa run output
available to me (read below).
* Currently bodhi behaves differently than before. For example the
two package sets I'm worried about right now are fail2ban and
mediawiki. ATM all packages are in rawhide and testing (f2b for
f15 is even stable), so I would expect at least the next-highest
Fedora packages to pass the autoqa. Still, trying to push any of
these packages from testing to stable I get:
* "This update has not yet met the minimum testing
requirements defined in the Package Update Acceptance
Criteria"
* The package request field in bodhi is empty, e.g. for he
packager at least it looks like no push attempt was ever
made, I don't know how it looks on the push granting
side, but I assume it looks the same.
* In some cases I don't get any indication why the autoqa
failed. For example for the fail2ban packages I either
received positive autoqa results or none at all (for f14
I never received any autoqa). Still the fail2ban
packages are rejected (all but f15).
* I also never received any autoqa info on
mediawiki-1.16.4-57.fc15 even though bodhi/autoqa
rejects it.
* Most of the autoqa bits are accessible only through my mail
folder. Given that some autoqa never made it there (the assumed
fail2ban autoqa failures), nor the bodhi comments, I am
currently at a loss what to do with the fail2ban packages.

I like the general idea of an automated test suite, but currently autoqa
is blocking/rejecting package pushes on a.f.o/updates, or at least bodhi
tells me so. And the reasons for doing so are non transparent.

Focusing on fail2ban, which is the more traditional update from the two
(mediawiki was fasttracked by another update, so let's keep this example
simpler by ignoring it):

* The packages were submitted to testing, accepted and stayed there for
a while.
* packages pushed to stable are being rejected for f13/f14 w/o the
packager understanding why (there is no (f14) or just positive
(f13/f15) autoqa feedback), especially when f15 did get pushed
through.
* My push request seem to get currently dropped, so my packages remain
in a non-requesting testing-updates limbo, and probably no one
notices.

Please help with these packages - it takes me much longer to get them
through bodhi than the actual packaging/updating/testing of the
packages.

Also please consider the following for autoqa:

* make a firm decision on whether, when and how autoqa will enforce its
results. It isn't just informative ATM.
* place ALL autoqa results in bodhi, even repeated ones. Hunting the
autoqa output though mails and bodhi isn't helping.
* Make autoqa always informative. If autoqa fails allow the packager to
add a comment to the update for the pusher to evaluate whether the
packages should be pushed regardless of the autoqa failures. Please
don't automatically reset push requests!

Thanks!
--
http://thimm.gr/ - http://ATrpms.net/
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 08:44 AM
Axel Thimm
 
Default AutoQA: distro congestion?

On Sun, 2011-04-17 at 21:19 -0600, Tim Flink wrote:
> On 04/16/2011 01:52 PM, Axel Thimm wrote:
> Others have already commented on this, but I wanted to re-state that
> AutoQA is only in an informative mode right now. Any bodhi comments made
> by AutoQA have no karma and are not used for anything other than
> information. This will change sometime in the future, but not until at
> least the F16 timeframe.

See my post from a few minutes ago: 5 packages are currently rejected on
a "mark as stable" submission leaving the request field empty (as if I
hadn't tried to push) with the comment that the packages do not pass
autoqa.

> I took a look at your packages in bodhi [1] and was only able to find
> one update with FAILED comments from AutoQA. Are there updates that I'm
> missing here?

This is probably part of the problem, I have been trying to push all 5
packages that are now in testing with bodhi rejecting due to autoqa.
Even packages that do have a positive autoqa tag on them like
fail2ban-0.8.4-27.fc13.

> Looking at the depcheck output for mediawiki-1.16.4-57.fc13, however, I see:
>
> mediawiki-nomath-1.16.4-57.fc13.x86_64 from pending has depsolving problems
> --> mediawiki-nomath conflicts with php-common
>
> This stems from the addition of the "Conflicts: php-common = 5.3.1" that
> was added to the spec file [2] for 1.16.2-56.

What about f14 and f15? At least for f15 autoqa should have greenlit the
package, yet it never even tested the package, but bodhi calims it fails
autoqa. f14 should had never passed before f15, but it still did, but is
blocked anyway (???).

> PS - you will generally get a faster response to AutoQA related
> questions by posting to autoqa-devel@lists.fedorahosted.org instead of
> just devel@

Thanks, but I just want to get the packages through, I'm on several
dozen fedora lists already and cannot process the information stream in
time. I also consider autoqa issues - if they are stalling packages as I
experience - important for anyone in devel. Alone the fact that everyone
here thinks differently about how autoqa works or should work shows that
the discussion is helpful on this list.
--
http://thimm.gr/ - http://ATrpms.net/
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 08:50 AM
Kevin Kofler
 
Default AutoQA: distro congestion?

Axel Thimm wrote:
> Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
> prominently displays:
>
> Bodhi is now enforcing the Package Update Acceptance
> Criteria across all Fedora releases.

The criteria which are being enforced do not include AutoQA results at this
time. They do, however, include minimum testing (time and/or karma)
requirements, which probably explains why your stable request was rejected.
(And I've been fighting against those requirements since they were first
proposed, because I strongly believe this decision should really be up to
the maintainer, but I lost that battle.)

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 08:54 AM
Sven Lankes
 
Default AutoQA: distro congestion?

On Tue, Apr 19, 2011 at 11:44:03AM +0300, Axel Thimm wrote:

> This is probably part of the problem, I have been trying to push all 5
> packages that are now in testing with bodhi rejecting due to autoqa.
> Even packages that do have a positive autoqa tag on them like
> fail2ban-0.8.4-27.fc13.

According to the Bodhi-Status-Site, you unpushed the update on
2011-04-17 21:24:29, then submitted it again a few seconds later.

It has been pushed to testing on 2011-04-17 21:24:29 and it now needs to
stay in testing for a week (or until it has reached sufficient karma
including proventester feedback) until it can be pushed to stable.

This is what bodhi refers to with "Bodhi is now enforcing the Package
Update Acceptance Criteria across all Fedora releases." - that text also
links to https://fedoraproject.org/wiki/Package_update_acceptance_criteria

--
sven === jabber/xmpp: sven@lankes.net
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 09:05 AM
Michael Schwendt
 
Default AutoQA: distro congestion?

On Tue, 19 Apr 2011 11:30:44 +0300, AT wrote:

> > AutoQA is still in testing phase, there is no enforcement yet.
>
> Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
> prominently displays:
>
> Bodhi is now enforcing the Package Update Acceptance
> Criteria across all Fedora releases.

This links to the Wiki and explains what is being enforced. It is much
to read, but in short,

https://fedoraproject.org/wiki/Package_update_acceptance_criteria#All_other_updat es

your packages (F-14, F-13) either need to sit for one week in testing
or reach the karma threshold you configured. Mediawiki updates have been
submitted on 04-16, so the one week is not over yet. For F-15 it's three
days only currently. Bodhi adds a comment to a ticket when the acceptance
criteria are satisfied and when the minimum time in testing has been
reached. (It adds that comment even if the ticket has reached a negative
karma value meanwhile, so be warned.)

Some packagers have been observed circumventing the system by configuring
a karma threshold of 1, so their own +1 vote or the first one from an arbitrary
tester make it possible to mark the update stable.

> And the conversion to testing was mentioned when I tried to push the
> package, not later, when a human could process the request.

You've tried to select "stable" as the target already when submitting
the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki,
why didn't you choose "security" instead of "stable"?

> Currently the autoqa process is very muddy to me and probably other
> packagers as well:

The AutoQA stuff that's been running lately does not add to the Package Update
Acceptance criteria yet. It would need to either spend karma points to inform
bodhi or even receive special veto privileges to influence bodhi's decision on
whether an update may enter the stable repo.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 02:07 PM
James Laska
 
Default AutoQA: distro congestion?

On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
> On Tue, 19 Apr 2011 11:30:44 +0300, AT wrote:
>
> > > AutoQA is still in testing phase, there is no enforcement yet.
> >
> > Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
> > prominently displays:
> >
> > Bodhi is now enforcing the Package Update Acceptance
> > Criteria across all Fedora releases.
>
> This links to the Wiki and explains what is being enforced. It is much
> to read, but in short,
>
> https://fedoraproject.org/wiki/Package_update_acceptance_criteria#All_other_updat es
>
> your packages (F-14, F-13) either need to sit for one week in testing
> or reach the karma threshold you configured. Mediawiki updates have been
> submitted on 04-16, so the one week is not over yet. For F-15 it's three
> days only currently. Bodhi adds a comment to a ticket when the acceptance
> criteria are satisfied and when the minimum time in testing has been
> reached. (It adds that comment even if the ticket has reached a negative
> karma value meanwhile, so be warned.)
>
> Some packagers have been observed circumventing the system by configuring
> a karma threshold of 1, so their own +1 vote or the first one from an arbitrary
> tester make it possible to mark the update stable.
>
> > And the conversion to testing was mentioned when I tried to push the
> > package, not later, when a human could process the request.
>
> You've tried to select "stable" as the target already when submitting
> the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki,
> why didn't you choose "security" instead of "stable"?
>
> > Currently the autoqa process is very muddy to me and probably other
> > packagers as well:
>
> The AutoQA stuff that's been running lately does not add to the Package Update
> Acceptance criteria yet. It would need to either spend karma points to inform
> bodhi or even receive special veto privileges to influence bodhi's decision on
> whether an update may enter the stable repo.

Right, once we iron out any kinks with bodhi tagging and autoqa tests
+scheduling, we'll revisit better ways to convey automated test results
in a non-irritating manner.

To reiterate again, AutoQA simply adds a comment to the bodhi update
informing the maintainer about automated test results for packages in
that update. It does not add karma (positive or negative) in any way at
this time. If the results don't make sense or are wrong, please stop by
autoqa-devel@lists.fedorahosted.org and let folks know.

Thanks,
James
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 03:37 PM
Adam Williamson
 
Default AutoQA: distro congestion?

On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:

> Some packagers have been observed circumventing the system by configuring
> a karma threshold of 1, so their own +1 vote or the first one from an arbitrary
> tester make it possible to mark the update stable.

Not...really. The update submitter's own vote should count as 0 (I can't
remember if this has landed in current Bodhi yet). Setting the karma
threshold to 1 cannot circumvent the 'proventesters +1 and any +1'
requirement for stable releases; just try it, it doesn't work. The
correct requirements are enforced whatever you set the autopush
threshold to.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 03:39 PM
Adam Williamson
 
Default AutoQA: distro congestion?

On Tue, 2011-04-19 at 11:30 +0300, Axel Thimm wrote:
> On Sun, 2011-04-17 at 08:01 +0200, Kevin Kofler wrote:
> > Axel Thimm wrote:
> > > Are you sure? I requested a push of the packages to stable (some were in
> > > testing for a week, others were security updates) and the message was
> > > that it doesn't pass AutoQA, so it converted to request to push only to
> > > testing and indeed bodhi has marked the request as to testing only.
> >
> > It did that because you changed it.
> >
> > If you had kept the request to stable, it would have been pushed to stable.
> >
> > AutoQA is still in testing phase, there is no enforcement yet.
>
> Maybe the bodhi messages confused me. When I logon to a.f.o/updates it
> prominently displays:
>
> Bodhi is now enforcing the Package Update Acceptance
> Criteria across all Fedora releases.

One thing to clarify in case it's not sufficiently clear from all the
other replies: this message is *not new* and has nothing to do with
AutoQA at all, and specifically nothing to do with the recent
implementation of advisory AutoQA upgrade path checking. This message
has been there for several months, ever since we started enforcing karma
requirements, and is nothing to do with AutoQA. Note that the message
doesn't *say* anything about AutoQA.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-19-2011, 04:30 PM
Thomas Spura
 
Default AutoQA: distro congestion?

On Tue, 19 Apr 2011 08:37:25 -0700
Adam Williamson wrote:

> On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
>
> > Some packagers have been observed circumventing the system by
> > configuring a karma threshold of 1, so their own +1 vote or the
> > first one from an arbitrary tester make it possible to mark the
> > update stable.
>
> Not...really. The update submitter's own vote should count as 0 (I
> can't remember if this has landed in current Bodhi yet). Setting the
> karma threshold to 1 cannot circumvent the 'proventesters +1 and any
> +1' requirement for stable releases; just try it, it doesn't work. The
> correct requirements are enforced whatever you set the autopush
> threshold to.

I just tried it (ok after the threshold of 3 days in f15) and it worked:
https://admin.fedoraproject.org/updates/ipython-0.10.2-1.fc15

Thomas
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 04:57 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org