FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Advisory Board

 
 
LinkBack Thread Tools
 
Old 04-02-2008, 05:55 PM
"Jon Stanley"
 
Default Spins proposals

On Wed, Apr 2, 2008 at 1:06 PM, Jeff Spaleta <jspaleta@gmail.com> wrote:

> I will be blunt. I think we've had a significant problem with the QA
> process for multiple releases now. We must find a way to organize
> volunteer labor better with regard to QA..generally.

A lot of that is being taken care of with F9 - we had (IMHO) a very
successful QA process for Beta. There is now a testing matrix that
includes owners, test cases, etc, that we've not really had before.

> If QA wants to officially get on the record as saying they aren't going to
> take the responsibility of testing the composed images that RelEng has
> selected and working on as part of a release cycle... then by all means get
> on the record....as a group put your foot down in protest...so I can come in
> with my riot gear and tear gas and bust some skulls.

Well, I don't think I can speak for QA (Will), but I'm fairly certain
that we don't have the resources to do this. We have a difficult time
as is doing the release testing that exists today, let alone adding an
as yet undetermined number of custom spins to the mix.

Spin owners are responsible for building, maintaining, and executing
their own test plans. These plans should include the standard Fedora
test plan, and any extensions the spin owners feel appropriate.

> Or QA as a group try to get involved with the formation of the Spin SIG and
> make sure they are well prepared to do a significant amount of the QA as
> part of Kickstart Pool management.

We can try to do this, but our resources are stretched pretty thin
right now. As mentioned before, spin owners are responsible for the
QA of their spins. We (QA) cannot do much more than basic sanity
tests.

I am now prepared for the riot gear and tear gas.

-Jon

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-02-2008, 06:09 PM
"Jeff Spaleta"
 
Default Spins proposals

On Wed, Apr 2, 2008 at 9:55 AM, Jon Stanley <jonstanley@gmail.com> wrote:

We can try to do this, but our resources are stretched pretty thin

right now. *As mentioned before, spin owners are responsible for the

QA of their spins. *We (QA) cannot do much more than basic sanity

tests.



I am now prepared for the riot gear and tear gas.


Let me be clear. If you want to hand the testing right back over the the spin maintainer with guidance on what needs to be done....fine with me.* The Board isn't going to mandate how QA tasks its work. But this most definitely falls under QA to decide and guide others on what and how and who needs to do this or any release testing.* Deputize all the members of the Spin Sig if you have to, but its QA's call to organize the workflow for QA.** You've a broad mandate to organize the testing space.


-jef


_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-02-2008, 06:25 PM
"Jon Stanley"
 
Default Spins proposals

On Wed, Apr 2, 2008 at 2:09 PM, Jeff Spaleta <jspaleta@gmail.com> wrote:

> Let me be clear. If you want to hand the testing right back over the the
> spin maintainer with guidance on what needs to be done....fine with me.

OK, I was thinking that you were saying that the QA team had to
personally do all of this work that we have no resources to do, this
is more in line with what we had in mind - give a basic Fedora test
plan, and have the spin maintainer expand on/maintain/execute that
test plan.

> Board isn't going to mandate how QA tasks its work. But this most definitely
> falls under QA to decide and guide others on what and how and who needs to
> do this or any release testing.

Agreed.

> Deputize all the members of the Spin Sig if
> you have to, but its QA's call to organize the workflow for QA. You've a
> broad mandate to organize the testing space.

Yep, for this specific workflow, we're on the hook to provide a test
plan of things that have to work in anything that we call Fedora, and
then deputize the Spin SIG to actually execute the plan and maintain
it on an ongoing basis.

Sorry for any misunderstanding!

-Jon

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 03:44 PM
Will Woods
 
Default Spins proposals

On Wed, 2008-04-02 at 09:06 -0800, Jeff Spaleta wrote:

> * Fedora QA is tasked with testing all the release spins.
> Nominally being part of that team, I don't think QA has the
> resources (ie) enough people involved considering that we have
> a six month release cycle and lots of bugs to deal with
>
> How QA gets the testing done is QA's perview. But the tasking is most
> certaintly correctly scoped.
>
> I will be blunt. I think we've had a significant problem with the QA
> process for multiple releases now. We must find a way to organize
> volunteer labor better with regard to QA..generally.

And here we have a beautifully ironic illustration of the real problem
with QA:

It took me a day to respond to this email because I was too busy:
- running the QA meeting,
- triaging the F9Blocker/F9Tracker lists,
- retesting bugs that should be fixed,
- testing upgrades from F8,
- testing fixed builds for problems in rawhide, and
- filing bug reports for new problems found.

Get it? I'm too busy *doing* QA to *improve* QA. I'm too understaffed to
work on staffing up. Chicken, meet egg.

Now. If you'd care to expand on *your* thoughts on the problems with the
QA Process, I'm all ears.

> If QA wants to officially get on the record as saying they aren't
> going to take the responsibility of testing the composed images that
> RelEng has selected and working on as part of a release cycle... then
> by all means get on the record....as a group put your foot down in
> protest...so I can come in with my riot gear and tear gas and bust
> some skulls.

Seriously? That's how we're going to start the discussion? You're gonna
bust my head in unless I agree to take on *more* work when the entire
problem with QA is that I'm *already* too overworked to do anything?

Something is not quite right here.

> Or QA as a group try to get involved with the formation of the Spin
> SIG and make sure they are well prepared to do a significant amount of
> the QA as part of Kickstart Pool management.

Getting involved in the Spin SIG: yes.
Kickstart Pool management: Also yes.

Doing QA for every spin produced: Very no.

Here's how it's gonna happen: QA will provide the same plans,
guidelines, and tools that we use for Fedora now, and we'll provide
advice on using and improving them. But *the spins are responsible for
performing their own testing*.

QA already provides these things for the standard releases:
Release guidelines
http://fedoraproject.org/wiki/QA/ReleaseCriteria
A detailed installation test plan..
http://fedoraproject.org/wiki/QA/TestPlans/Fedora9Install
..composed of dozens of individual test cases..
http://fedoraproject.org/wiki/QA/TestCases
..along with templates for creating new test cases and test plans.
http://fedoraproject.org/wiki/QA/TestCases/TestCaseTemplate
http://fedoraproject.org/wiki/QA/TestPlans/TestPlanTemplate
And a test summary page where we track our testing results.
http://fedoraproject.org/wiki/QA/TestResults/TestSummaryTemplate

For each release, we triage bugs according to the ReleaseCriteria,
update the installation test cases to reflect new features, execute the
installation test cases, and track and report results.

The owners of each spin should be responsible for:
- Maintaining spin-specific ReleaseCriteria or TestCases, if needed
- Tracking spin-specific bugs,
- Executing test plans for their spin, and
- Tracking the results thereof.

Now - I'm not saying they have to do all the same work. The spin owners
can definitely *choose* to skip some testing, if they like. And I want
to avoid duplicate work - it's perfectly fine to assume that if
raid1-root works in the normal Fedora spin, it probably works in the
Electronics Lab spin as well.

And, really, all I'm asking is that each spin spend some time doing QA
on their stuff. It helps the overall test effort by making more official
"QA People", spreading the test load over all the spins, and will create
new test cases. And hopefully this will drive an effort to improve the
tools for managing and tracking test plans and execution.

But this is the important thing: THE SPIN OWNERS ARE ULTIMATELY
RESPONSIBLE FOR GETTING THEIR SPIN TESTED.

You can find your own QA guys, you can do the testing yourself,
whatever. But the current QA team is already far, far too busy to
actually sit down and test all the bits we produce *now*. We cannot -
and will not - test *new* spins that are dropped in our laps, so long as
that would take time away from testing Fedora proper.

This will probably remain the case until such time as we've managed to
build the robust automated testing infrastructure that Fedora deserves.

> The Spin SIG may very well find that its in their mutual best interest
> in getting involved more directly in the QA process..even before a
> spin has been selected for release. And it very well maybe in QA's
> best interest to find a way to guide the Spin SIGs involvement in
> post-release testing as well. But that's a conversation for the Spin
> SIG and QA to have at some point between now and the start of the F10
> release cycle.

Oh, definitely. I have a feeling the Spin SIG and QA and the Kickstart
Pool will all be very chummy in the near future.

Until then, though, we've got a release to work on.

-w

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 04:17 PM
"Stephen John Smoogen"
 
Default Spins proposals

On Thu, Apr 3, 2008 at 9:44 AM, Will Woods <wwoods@redhat.com> wrote:
> On Wed, 2008-04-02 at 09:06 -0800, Jeff Spaleta wrote:
>
> > * Fedora QA is tasked with testing all the release spins.
> > Nominally being part of that team, I don't think QA has the
> > resources (ie) enough people involved considering that we have
> > a six month release cycle and lots of bugs to deal with
> >
> > How QA gets the testing done is QA's perview. But the tasking is most
> > certaintly correctly scoped.
> >
> > I will be blunt. I think we've had a significant problem with the QA
> > process for multiple releases now. We must find a way to organize
> > volunteer labor better with regard to QA..generally.
>
> And here we have a beautifully ironic illustration of the real problem
> with QA:
>
> It took me a day to respond to this email because I was too busy:
> - running the QA meeting,
> - triaging the F9Blocker/F9Tracker lists,
> - retesting bugs that should be fixed,
> - testing upgrades from F8,
> - testing fixed builds for problems in rawhide, and
> - filing bug reports for new problems found.
>
> Get it? I'm too busy *doing* QA to *improve* QA. I'm too understaffed to
> work on staffing up. Chicken, meet egg.
>
> Now. If you'd care to expand on *your* thoughts on the problems with the
> QA Process, I'm all ears.
>
>

Long but important rant shortened... my views are the following:

1) QA is understaffed. While QA is always understaffed, it is
currently at Wanger levels of understaffed... the paid for QA people
are not able to think straight because they are always behind, always
getting more work, and deadlines are always there. Throwing more
people at this point won't help unless you are going to say its ok for
the current QA people to stop what they are doing and work on
improving things.

2) Release times for QA are always stressful because every not found
bug is their fault, and every found bug that delays things are their
fault. And at a certain point the stress is so high that even helping
hands look like more whips. [Just like a starving dog will bite a
rescuer's hand...]

3) QA only seems to get a focus at the worst times... when things are
hitting the fan. You want to improve QA, do it when there isn't a
release. The only times I have seen QA improve at companies is when
someone high enough says "This shit is not going to go on any more."
and people are beaten out of the habit of viewing QA as something that
comes at the end of a project. And its a beating that has to happen
quite a bit because well.. we always have something on our mind that
is more important (like getting a build system, a package, etc done)

4) If you are not seeing your QA people at meetings, at gatherings,
doing their weekly blog etc.. you need to stop whatever you think is
currently more important than QA and figure out where things are
broken. Otherwise you might as well just get rid of QA because they
are only being viewed as an expense at that point.

Grumpy old man smooge will now shut-up.


--
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 04:19 PM
"Jeff Spaleta"
 
Default Spins proposals

On Thu, Apr 3, 2008 at 7:44 AM, Will Woods <wwoods@redhat.com> wrote:
> Seriously? That's how we're going to start the discussion? You're gonna
> bust my head in unless I agree to take on *more* work when the entire
> problem with QA is that I'm *already* too overworked to do anything?
>
> Something is not quite right here.

I didn't say who's particular skull was at risk. I go out and wade
the the sea of uncontributing community and pound on them if need be.
If I had the ear of the CEO. I'd pound on him if I thought it would
help. But there is a problem here and I'm not afraid to look directly
into other people's brains in an effort to divine a solution. And now
that you are on record concerning being too overwhelmed to the build
where you can't build a QA community, then you've given the Board a
reason to get involved in figuring out how to unbreak what's going
wrong.

Here's the deal, I firmly believe that the Board shouldn't micromanage
a particular subgroup, even if that group isn't performing
'optimally', whatever that means. The group has to escalate to the
Board and demand that we get involved in their internal process.

> Here's how it's gonna happen: QA will provide the same plans,
> guidelines, and tools that we use for Fedora now, and we'll provide
> advice on using and improving them. But *the spins are responsible for
> performing their own testing*.

I'm fine with that. I'm fine with QA deciding to re-task testing,
because the decisions on what the how/who/when/where's for testing is
fundamentally a part of QA's role. I do not want to have the Board
setup frameworks that reach in and tell a group how they are going to
do things unnecessarily.

> And, really, all I'm asking is that each spin spend some time doing QA
> on their stuff.

You can be stronger than that... you could 'demand' it. When the Spin
SIG brings a spin to the Board for trademark approval, if they haven't
made any effort to run through a set of predefined tests as part of
the technical review, I'll be cracking their skulls. But the Board
will most likely ask QA to rubber stamp whatever testing process the
Spin SIG sets up as part of the technical review leading up to
trademark approval.

-jef

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 05:04 PM
"Tom "spot" Callaway"
 
Default Spins proposals

On Thu, 2008-04-03 at 10:17 -0600, Stephen John Smoogen wrote:
> Grumpy old man smooge will now shut-up.

I wonder whether more than a handful of folks will catch most of those
Red Hat Ancient History lessons. :/

~spot

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 07:21 PM
"Stephen John Smoogen"
 
Default Spins proposals

On Thu, Apr 3, 2008 at 11:04 AM, Tom spot Callaway <tcallawa@redhat.com> wrote:
> On Thu, 2008-04-03 at 10:17 -0600, Stephen John Smoogen wrote:
> > Grumpy old man smooge will now shut-up.
>
> I wonder whether more than a handful of folks will catch most of those
> Red Hat Ancient History lessons. :/
>

/me starts work on getting a Red Hat Department of History so we know
why we are making the same mistakes again...



--
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 07:33 PM
"Jeff Spaleta"
 
Default Spins proposals

On Thu, Apr 3, 2008 at 11:21 AM, Stephen John Smoogen <smooge@gmail.com> wrote:
> /me starts work on getting a Red Hat Department of History so we know
> why we are making the same mistakes again...

Uhm... I think for Fedora as a project, we could probably actually
start doing that for most Board level discussion since we are making
an effort to be transparent. Public meeting minutes help.. a lot.

-jef

_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 
Old 04-03-2008, 10:24 PM
Jesse Keating
 
Default Spins proposals

On Mon, 2008-03-31 at 09:23 -0400, Paul W. Frields wrote:
> Jeff Spaleta has been kind enough to post his spin proposals on the
> wiki
> for easier reading and comment. The main proposal is at:
>
> https://fedoraproject.org/wiki/JefSpaleta/SpinReleaseProcessProposal
>
> There are supplemental addenda at:
>
> https://fedoraproject.org/wiki/JefSpaleta/SpinReleaseProcessAmendments
> https://fedoraproject.org/wiki/JefSpaleta/CommunityHostedSpins

releng agrees with these proposals in principle. Josh Boyer will be
leading the effort to have releng provide the things these proposals lay
out as our responsibility, and participating in the spins SIG from a
releng point of view.

--
Jesse Keating
Fedora -- All my bits are free, are yours?
_______________________________________________
fedora-advisory-board mailing list
fedora-advisory-board@redhat.com
http://www.redhat.com/mailman/listinfo/fedora-advisory-board
 

Thread Tools




All times are GMT. The time now is 06:54 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org