Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Development (http://www.linux-archive.org/fedora-development/)
-   -   Provide more testing feedback (was: Refining the update queues/process) (http://www.linux-archive.org/fedora-development/335531-provide-more-testing-feedback-refining-update-queues-process.html)

Adam Williamson 03-03-2010 11:07 PM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, 2010-03-04 at 00:55 +0100, Till Maas wrote:

> > So - for the third time - a package being in updates-testing for a few
> > days and getting no negative feedback is a moderate strength indicator
> > that it's not egregiously broken. Not a super-strong indicator, but
> > better than a kick in the teeth.
>
> It probably only means that the meta-data of the installed package is
> not broken, but if they do not use all packages installed daily, then
> there is not much test coverage.

The types of breakage that most worry us are the ones where some update
causes really big and obvious problems that affect lots of people.
Happily, this is the kind of breakage you're most likely to get negative
feedback on when it happens. :)

So yes, the current process probably isn't very good at testing whether
a given update does absolutely everything it's supposed to do, in all
cases. It's not brilliant even at testing whether a given update works
at all, if that update is a fairly obscure package. What it _can_ do
reasonably well is catch the situation where an update mistakenly breaks
the world - where you install it and then suddenly you can't boot or
GNOME won't start or your network connection is broken or whatever. And
that's the kind of thing we're really trying to prevent.

I don't think a system where all updates had to stay in -testing for a
few days would catch all update problems. We'd still probably ship some
buggy updates. But hopefully we wouldn't again have the situation where
we're standing around scratching our heads and thinking 'how the *hell*
did that get shipped, when it breaks normal functionality for thousands
of people?'
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Till Maas 03-04-2010 01:32 AM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, Mar 04, 2010 at 12:55:46AM +0100, Till Maas wrote:

> So here is a first ugly script to easily give feedback for all installed
> testing updates that were created after a certain date (I did not find
> an easy way to get all testing updates, one did not yet comment on):
> http://till.fedorapeople.org/tmp/give-feedback.py
>
> It kind of works like this: Show the update, ask for karma to create a
> comment, otherwise it is skipped. Then a comment needs to be entered.
> Sometimes it will ask for the FAS password using the bodhi client.
>
> Update: rpmdevtools-7.8-1.fc12
> Comment? -1/0/1 ->karma, other -> skip> 1
> Comment> "rpmdev-newspec foo" still works
>
> The prompt has readline support, so if you just want to use the same
> comment again and again, you can use cursor-up to get the comment again.
>
> Btw. the script fails in case only subpackages of a package are
> installed. But if it is used, I will create a saner version.
> Nevertheless, there is also now the Fedora Engineering Service, so if
> you want to improve your testing tools, you might want to ask there.

A less ugly script can now be found here:
http://till.fedorapeople.org/tmp/easy-karma.py
Improvements:
- display update details, e.g. bugs and notes
- use src.rpm to find matching update
- skip updates that have already been commented

With this giving karma is so simple, that there is no excuse for not
doing so. I'll add an git repo and clean it up more the next days.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 03-04-2010 01:46 AM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, 2010-03-04 at 03:32 +0100, Till Maas wrote:

> A less ugly script can now be found here:
> http://till.fedorapeople.org/tmp/easy-karma.py
> Improvements:
> - display update details, e.g. bugs and notes
> - use src.rpm to find matching update
> - skip updates that have already been commented
>
> With this giving karma is so simple, that there is no excuse for not
> doing so. I'll add an git repo and clean it up more the next days.

This looks absolutely yummy, thank you Till! I'll play with it a bit and
promote it to the QA group.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Adam Williamson 03-04-2010 04:36 AM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, 2010-03-04 at 03:32 +0100, Till Maas wrote:

> A less ugly script can now be found here:
> http://till.fedorapeople.org/tmp/easy-karma.py
> Improvements:
> - display update details, e.g. bugs and notes
> - use src.rpm to find matching update
> - skip updates that have already been commented
>
> With this giving karma is so simple, that there is no excuse for not
> doing so. I'll add an git repo and clean it up more the next days.

completely awesome, just let me file a bunch of feedback I never would
have managed to get to before. great work, thank you.

small nit: if a single update has, say, three packages in it, the script
presents it for your feedback three times.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Kevin Fenzi 03-04-2010 04:41 AM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, 04 Mar 2010 03:32:13 +0100
Till Maas <opensource@till.name> wrote:

> A less ugly script can now be found here:
> http://till.fedorapeople.org/tmp/easy-karma.py
> Improvements:
> - display update details, e.g. bugs and notes
> - use src.rpm to find matching update
> - skip updates that have already been commented
>
> With this giving karma is so simple, that there is no excuse for not
> doing so. I'll add an git repo and clean it up more the next days.

Excellent.

I just added 11 bits of karma to various packages I have been using for
a while but never got around to tracking down to give karma to. ;)

Perhaps this could be added into fedora-packager?

Nice work.

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Seth Vidal 03-04-2010 05:23 AM

Provide more testing feedback (was: Refining the update queues/process)
 
Till Maas <opensource@till.name> wrote:
>
>> A less ugly script can now be found here:
>> http://till.fedorapeople.org/tmp/easy-karma.py
>> Improvements:
>> - display update details, e.g. bugs and notes
>> - use src.rpm to find matching update
>> - skip updates that have already been commented
>>
>> With this giving karma is so simple, that there is no excuse for not
>> doing so. I'll add an git repo and clean it up more the next days.


Till,

Great script here's a small set of changes to have easy-karma use yum as a module
instead of via subprocess.

http://skvidal.fedorapeople.org/misc/easy-karma-yum.patch

if I get a chance I'll see if I can figure out how to use the bodhi module
at the end so it doesn't have to use any subprocess calls at all.

thanks
-sv

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Till Maas 03-04-2010 09:02 AM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, Mar 04, 2010 at 01:23:30AM -0500, Seth Vidal wrote:

> Great script here's a small set of changes to have easy-karma use yum as a module
> instead of via subprocess.
>
> http://skvidal.fedorapeople.org/misc/easy-karma-yum.patch

Thank you, I will integrate it later today, when I set up a git repo. Is
there a way to restrict the yum results using the time the packages were
installed?

> if I get a chance I'll see if I can figure out how to use the bodhi module
> at the end so it doesn't have to use any subprocess calls at all.

This is pretty straight forward, I will do this later today, too.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Seth Vidal 03-04-2010 12:03 PM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, 4 Mar 2010, Till Maas wrote:

> On Thu, Mar 04, 2010 at 01:23:30AM -0500, Seth Vidal wrote:
>
>> Great script here's a small set of changes to have easy-karma use yum as a module
>> instead of via subprocess.
>>
>> http://skvidal.fedorapeople.org/misc/easy-karma-yum.patch
>
> Thank you, I will integrate it later today, when I set up a git repo. Is
> there a way to restrict the yum results using the time the packages were
> installed?

I can't query based on it - but it's easy for comparison purposes to look
up the installed time from the package object.

the value is pkg_obj.installtime

it'll return the time in seconds

>> if I get a chance I'll see if I can figure out how to use the bodhi module
>> at the end so it doesn't have to use any subprocess calls at all.
>
> This is pretty straight forward, I will do this later today, too.

great.
-sv

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Kevin Kofler 03-04-2010 12:40 PM

Provide more testing feedback (was: Refining the update queues/process)
 
Kevin Fenzi wrote:
> Perhaps this could be added into fedora-packager?

Well, it's useful also for testers (or even just users) who are not
packagers, so I'm not sure that's the best place.

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Till Maas 03-04-2010 12:59 PM

Provide more testing feedback (was: Refining the update queues/process)
 
On Thu, Mar 04, 2010 at 01:23:30AM -0500, Seth Vidal wrote:

> Great script here's a small set of changes to have easy-karma use yum as a module
> instead of via subprocess.
>
> http://skvidal.fedorapeople.org/misc/easy-karma-yum.patch

There is now a git repo and your patch is included:
http://fedorapeople.org/gitweb?p=till/public_git/fedora-easy-karma.git;a=summary

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel


All times are GMT. The time now is 01:18 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.