FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 03-03-2010, 06:18 PM
Till Maas
 
Default Refining the update queues/process

On Wed, Mar 03, 2010 at 11:07:27AM -0500, Seth Vidal wrote:
>
>
> On Wed, 3 Mar 2010, Till Maas wrote:
>
> > On Wed, Mar 03, 2010 at 08:42:57AM -0500, Seth Vidal wrote:
> >
> >> On Wed, 3 Mar 2010, Till Maas wrote:
> >
> >>> Are there even any metrics about how many bad updates happened? For me
> >>> bug that can be fixed issuing an update are a lot more than regressions
> >>> with updates or new bugs introduced with updates. If updates are slowed
> >>> down, this will get even worse. Especially because the proposal is to
> >>> use time instead of test coverage as the criterion to push an update to
> >>> stable.
> >>
> >> Actually the proposal is time AND test coverage.
> >
> > I mind have misunderstood it, but afaics it only says that it will be
> > tested, because it spent time in updates-testing, but this is not even
> > true nowadays, even if packages stay long in updates-testing.
>
> Having more time opens us up to more testing days and in the near future
> autoqa to help us bounce obviously bad things.

This statement fails to address that packages that stays long in
updates-testing are subject to testing. Btw. I am also pretty sure that
most of the manual testing time is better spent writing automated tests,
unless you consider just updating to updates-testing and see if
something bad happens sufficient testing. But this is not even enough to
find regressions, because one needs to investigate how to reproduce it
and then also test the update/release version of the package to know,
whether it is a regression or just a bug that was not triggered before.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-03-2010, 09:06 PM
Adam Williamson
 
Default Refining the update queues/process

On Wed, 2010-03-03 at 17:04 +0100, Till Maas wrote:

> I mind have misunderstood it, but afaics it only says that it will be
> tested, because it spent time in updates-testing, but this is not even
> true nowadays, even if packages stay long in updates-testing.

as we've explained several times, most packages that go to
updates-testing for a few days *are* being tested, even if they get no
apparent Bodhi feedback. Several QA group members run with
updates-testing enabled and so get all packages (that they have
installed) which go through updates-testing. They do not file positive
feedback for every single package because there's just too many, but if
they notice breakage, they file negative feedback.

So - for the third time - a package being in updates-testing for a few
days and getting no negative feedback is a moderate strength indicator
that it's not egregiously broken. Not a super-strong indicator, but
better than a kick in the teeth.

This is why what winds up getting proposed to FESco is probably going to
be something along the lines of *either* acquiring a certain level of
positive feedback *or* sitting in testing for a few days without
acquiring any negative feedback. So you can either submit your update
and wait a few days to push it, or submit it and ask a couple of people
to test it and post positive feedback, and then you'll be able to push
it immediately.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 12:38 PM
Michael Schwendt
 
Default Refining the update queues/process

On Wed, 03 Mar 2010 14:06:33 -0800, Adam wrote:

> as we've explained several times,

It won't get more correct by simply repeating it over and over again.

> most packages that go to
> updates-testing for a few days *are* being tested, even if they get no
> apparent Bodhi feedback.

Certainly not most packages. Just those "Core"-type of packages whose
features cannot be avoided at run-time, such as essential system programs
that are executed always and on every installation. Or popular apps that
are *really* *really* widely and very actively used with many of their
features.

> Several QA group members run with
> updates-testing enabled and so get all packages (that they have
> installed)

Which is not equal to "most packages".

> which go through updates-testing. They do not file positive
> feedback for every single package because there's just too many, but if
> they notice breakage, they file negative feedback.

And they simply don't and can't notice all bugs and regressions. Audacious
2.1 in F12 development apparently hasn't seen real testing before F12 was
released. Since then, bug reports have been flowing in. Same with
Audacious 2.2 that became sort of a mandatory upgrade, so I could reduce
the patch count. Only after it had been released as stable update, the bug
reporting started again.

Too few users have updates-testing enabled. Too few bug reporters are
brave enough to enable updates-testing for a bug-fix referred to in
bugzilla.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 03:19 PM
Adam Williamson
 
Default Refining the update queues/process

On Fri, 2010-03-05 at 14:38 +0100, Michael Schwendt wrote:

> > which go through updates-testing. They do not file positive
> > feedback for every single package because there's just too many, but if
> > they notice breakage, they file negative feedback.
>
> And they simply don't and can't notice all bugs and regressions. Audacious
> 2.1 in F12 development apparently hasn't seen real testing before F12 was
> released. Since then, bug reports have been flowing in. Same with
> Audacious 2.2 that became sort of a mandatory upgrade, so I could reduce
> the patch count. Only after it had been released as stable update, the bug
> reporting started again.
>
> Too few users have updates-testing enabled. Too few bug reporters are
> brave enough to enable updates-testing for a bug-fix referred to in
> bugzilla.

Thank you for the very selective quoting, wherein you carefully cut out
all the bits where I explicitly acknowledged that the system does not
catch all problems, and painstaking explained that this is not what we
expect it to do, nor was anyone assuming that it did when the proposal
to require packages go through updates-testing was made. That's a great
way to have a productive discussion.

*sigh*
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:01 PM
Michael Schwendt
 
Default Refining the update queues/process

On Fri, 05 Mar 2010 08:19:25 -0800, Adam wrote:

> On Fri, 2010-03-05 at 14:38 +0100, Michael Schwendt wrote:
>
> > > which go through updates-testing. They do not file positive
> > > feedback for every single package because there's just too many, but if
> > > they notice breakage, they file negative feedback.
> >
> > And they simply don't and can't notice all bugs and regressions. Audacious
> > 2.1 in F12 development apparently hasn't seen real testing before F12 was
> > released. Since then, bug reports have been flowing in. Same with
> > Audacious 2.2 that became sort of a mandatory upgrade, so I could reduce
> > the patch count. Only after it had been released as stable update, the bug
> > reporting started again.
> >
> > Too few users have updates-testing enabled. Too few bug reporters are
> > brave enough to enable updates-testing for a bug-fix referred to in
> > bugzilla.
>
> Thank you for the very selective quoting, wherein you carefully cut out
> all the bits where I explicitly acknowledged that the system does not
> catch all problems, and painstaking explained that this is not what we
> expect it to do, nor was anyone assuming that it did when the proposal
> to require packages go through updates-testing was made. That's a great
> way to have a productive discussion.
>
> *sigh*

You can get a full quote:

| as we've explained several times, most packages that go to
| updates-testing for a few days *are* being tested, even if they get no
| apparent Bodhi feedback. Several QA group members run with
| updates-testing enabled and so get all packages (that they have
| installed) which go through updates-testing. They do not file positive
| feedback for every single package because there's just too many, but if
| they notice breakage, they file negative feedback.
|
| So - for the third time - a package being in updates-testing for a few
| days and getting no negative feedback is a moderate strength indicator
| that it's not egregiously broken. Not a super-strong indicator, but
| better than a kick in the teeth.
|
| This is why what winds up getting proposed to FESco is probably going to
| be something along the lines of *either* acquiring a certain level of
| positive feedback *or* sitting in testing for a few days without
| acquiring any negative feedback. So you can either submit your update
| and wait a few days to push it, or submit it and ask a couple of people
| to test it and post positive feedback, and then you'll be able to push
| it immediately.

It doesn't change anything, though. No feedback => nothing to rely on.
These recent discussions on this list could have been fruitful, btw.
For some people it has become a game of "I'm right - you aren't",
unfortunately.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:11 PM
Adam Williamson
 
Default Refining the update queues/process

On Fri, 2010-03-05 at 18:01 +0100, Michael Schwendt wrote:

> It doesn't change anything, though. No feedback => nothing to rely on.
> These recent discussions on this list could have been fruitful, btw.
> For some people it has become a game of "I'm right - you aren't",
> unfortunately.

Nothing like that. It just frustrates me when people don't debate
correctly. (Yes, there is a correct way to debate - take a philosophy
class). Replying to my post in which I said something, cutting out the
bit where I said it, and then saying the same thing as if it contradicts
what my post said - what does that achieve, exactly? Again, I already
acknowledged that the current process does not notice all problems in
candidate updates. All I've said is that it _does_ notice some problems,
and the fact that an update has 0 feedback is not an indication that
no-one has tried it and checked it doesn't make their system blow up.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:26 PM
Michael Schwendt
 
Default Refining the update queues/process

On Fri, 05 Mar 2010 09:11:10 -0800, Adam wrote:

> On Fri, 2010-03-05 at 18:01 +0100, Michael Schwendt wrote:
>
> > It doesn't change anything, though. No feedback => nothing to rely on.
> > These recent discussions on this list could have been fruitful, btw.
> > For some people it has become a game of "I'm right - you aren't",
> > unfortunately.
>
> Nothing like that. It just frustrates me when people don't debate
> correctly.

Then consider stopping to send further replies. You -- and some other
participants in these threads -- pipe out way too many replies in
quick succession.

> (Yes, there is a correct way to debate - take a philosophy class).

That's an impertinent comment, nothing else.

> Replying to my post in which I said something, cutting out the
> bit where I said it, and then saying the same thing as if it contradicts
> what my post said - what does that achieve, exactly?

It gives enough context.

> Again, I already
> acknowledged that the current process does not notice all problems in
> candidate updates.

No, not in a clear way. Instead, you keep emphasising that no negative
feedback is not equal to a package not having been tested at all. That's
just plain useless. Not even all broken deps are reported in bodhi.

> All I've said is that it _does_ notice some problems,
> and the fact that an update has 0 feedback is not an indication that
> no-one has tried it and checked it doesn't make their system blow up.

Refer to my previous reply for the comment on this where you insist on
repeating the same thing over and over again.

No further comment.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 04:33 PM
Adam Williamson
 
Default Refining the update queues/process

On Fri, 2010-03-05 at 18:26 +0100, Michael Schwendt wrote:

> > Nothing like that. It just frustrates me when people don't debate
> > correctly.
>
> Then consider stopping to send further replies. You -- and some other
> participants in these threads -- pipe out way too many replies in
> quick succession.

I wake up, read the list, post my replies and then do something else.
Naturally they arrive in groups.

> No, not in a clear way. Instead, you keep emphasising that no negative
> feedback is not equal to a package not having been tested at all. That's
> just plain useless. Not even all broken deps are reported in bodhi.

Why do you keep talking about 'all', as if the condition for success is
catching 'all' errors? No testing process catches all errors, people
aren't perfect. By your criteria, all testing is useless.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 08:16 PM
Michael Schwendt
 
Default Refining the update queues/process

On Fri, 05 Mar 2010 09:33:02 -0800, Adam wrote:

> > No, not in a clear way. Instead, you keep emphasising that no negative
> > feedback is not equal to a package not having been tested at all. That's
> > just plain useless. Not even all broken deps are reported in bodhi.
>
> Why do you keep talking about 'all', as if the condition for success is
> catching 'all' errors?

That is your claim.

In my comments it isn't universal quantification, but existential
quantification (∃). There is an update, which is still without feedback
after two weeks, and I cannot conclude anything about how much testing it
may have seen. That's very different from your "[...] most packages that
go to updates-testing for a few days *are* being tested, even if they get
no apparent Bodhi feedback. [...]"

> No testing process catches all errors, people
> aren't perfect. By your criteria, all testing is useless.

Not all. Twisting words isn't helpful. Trying to discuss with you is a
lost cause, unfortunately. It is my strong impression that you know
nothing about my point of view with regard to updates-testing. Perhaps
you're chasing ghosts or something. And in these huge threads it is too
late (and a waste of time) to even try to explain to individuals what
I think about updates-testing.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 08:46 PM
Adam Williamson
 
Default Refining the update queues/process

On Fri, 2010-03-05 at 22:16 +0100, Michael Schwendt wrote:
> On Fri, 05 Mar 2010 09:33:02 -0800, Adam wrote:
>
> > > No, not in a clear way. Instead, you keep emphasising that no negative
> > > feedback is not equal to a package not having been tested at all. That's
> > > just plain useless. Not even all broken deps are reported in bodhi.
> >
> > Why do you keep talking about 'all', as if the condition for success is
> > catching 'all' errors?
>
> That is your claim.
>
> In my comments it isn't universal quantification, but existential
> quantification (∃). There is an update, which is still without feedback
> after two weeks, and I cannot conclude anything about how much testing it
> may have seen. That's very different from your "[...] most packages that
> go to updates-testing for a few days *are* being tested, even if they get
> no apparent Bodhi feedback. [...]"

Ah. You're looking at it on a kind of micro level; 'how can I tell this
package has been tested?'

Maybe it makes it clearer if I explain more clearly that that's not
exactly how I look at it, nor (I think) how the rest of QA sees it, or
what the proposal to require -testing is intended to achieve. We're
thinking more about 'the big picture', and we're specifically thinking
about - as I said before - the real brown-paper-bag,
oh-my-god-what-were-they-thinking kinds of regressions, the 'systems
don't boot any more', 'Firefox doesn't run' kinds of forehead-slappers.
What we believe is that requiring packages to go to updates-testing for
some time improves our chances of avoiding that kind of issue.

Obviously, the more testing gets done in updates-testing, the better.
Hopefully Till's script will help a lot with that, it's already had a
very positive response. But the initial trigger for the very first
proposal from which all this discussion sprang was wondering what we
could do to avoid the really-big-duh kind of problem.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 02:59 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org