On Fri, 2010-03-05 at 23:52 +0100, Michael Schwendt wrote:
> On Fri, 05 Mar 2010 13:46:34 -0800, Adam wrote:
> > Ah. You're looking at it on a kind of micro level; 'how can I tell this
> > package has been tested?'
> Exactly. Because I don't like to act on assumptions.
> And "zero feedback" is only an indicator for "doesn't break badly", if
> there are N>1 testers with N>1 different h/w and s/w setups who have
> installed the update actually and have not rolled back without reporting a
> problem. This may apply to certain core packages, but _not_ to all pkgs.
I did say it was only a medium-strength indicator (in most cases), not
an infallible one, which was kinda intended to cover the above. IOW, I
agree, mostly. The more people we have running updates-testing, the more
likely we are to catch big breakages, of course.
> It takes days for updates to be distributed to mirrors. A week may be
> nothing for that important power-user of app 'A', who would find a problem
> as soon as he *would* try out a test-update.
In my experience, I get testing updates only a few hours after the email
listing them hits the mailing lists.
> Further, I hear about users who have run into problems with Fedora but
> haven't reported a single bug before. ABRT may help with that, but they
> would still need to create a bugzilla account, which is something they
> haven't done before and maybe won't do. Only sometimes a problem annoys
> them for so long that they see themselves forced to look into how to
> report a bug.
I'd hope this wouldn't describe anyone who takes the trouble to manually
activate updates-testing, but of course I could be wrong
> > Maybe it makes it clearer if I explain more clearly that that's not
> > exactly how I look at it, nor (I think) how the rest of QA sees it, or
> > what the proposal to require -testing is intended to achieve. We're
> > thinking more about 'the big picture', and we're specifically thinking
> > about - as I said before - the real brown-paper-bag,
> > oh-my-god-what-were-they-thinking kinds of regressions, the 'systems
> > don't boot any more', 'Firefox doesn't run' kinds of forehead-slappers.
> > What we believe is that requiring packages to go to updates-testing for
> > some time improves our chances of avoiding that kind of issue.
> The key questions are still: Which [special] packages do you want to cover?
> CRITPATH only? Or arbitrarily enforced delays for all packages?
The initial proposal to FESco would cover all packages. There is a
reason to cover all packages, which is that there _are_ cases where
there can be really serious breakage created by a package which isn't in
CRITPATH, though you could argue that's sufficiently unlikely to not
warrant holding up non-critpath packages. It could do with more
discussion, I guess.
> For example, it would make sense to keep those packages in updates-testing
> for an extended period, which have received feedback in bodhi _before_
> and which have a high bug reporting activity in bugzilla.
I'd say it's almost the opposite - you could hold those packages up only
for a little while, because you can be reasonably confident you'll find
out if they're badly broken *really fast*
Obviously, it's a tricky
> > Obviously, the more testing gets done in updates-testing, the better.
> > Hopefully Till's script will help a lot with that, it's already had a
> > very positive response. But the initial trigger for the very first
> > proposal from which all this discussion sprang was wondering what we
> > could do to avoid the really-big-duh kind of problem.
> I cannot answer that. Especially not because a package that may work fine
> for you and other testers, may be a really-big-duh for other users.
> This also leads to a not so funny scenario, where the big-duh has not
> been noticed by any tester during F-N development, but shortly after
> release it is found by ordinary users.
> When I give +1 karma, I either acknowledge only the fix for a specific bug
> that's linked, or I mention the type of usage, e.g. "basic daily usage" or
> "didn't try the new features". To not give a false impression that I may
> have tested everything. In general I hope that the feedback about me using
> the software is more helpful than zero feedback. However, it may still be
> that a certain feature/plugin I don't use is broken badly. That's not a
> guess, it has happened before and will happen again. With updates or shortly
> after a new Fedora release.
Yeah, this is a definite problem with the Bodhi system: it's not
particularly clear what +1 means or what it should mean, and different
reporters use it differently. It's definitely not something we've nailed
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
devel mailing list