FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 12-21-2010, 04:11 PM
Adam Williamson
 
Default Package-specific test case and critical path test case project: drafts for review

Hi, everyone. So, in the recent debate about the update process it again
became clear that we were lacking a good process for providing
package-specific test instructions, and particularly specific
instructions for testing critical path functions.

I've been working on a process for this, and now have two draft Wiki
pages up for review which together describe it:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_test_case_creation
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation

the first isn't particularly specific to this, but it was a prerequisite
that I discovered was missing: it's a guide to test case creation in
general, explaining the actual practical process of how you create a
test case, and the best principles to consider in doing it.

The second is what's really specific to this subject. It describes how
to create a set of test cases for a particular package, and a proposed
standardized categorization scheme which will allow us to denote test
cases as being associated with specific packages, and also denote them
as concerning critical path functionality.

Given that mediawiki has a handy API which also allows you to deal with
categories, this should make it easy to both manually and
programmatically derive a list of test cases for a given package, and a
list of *critical path* test cases for a given package. You can do this
manually, but I also envision Bodhi and fedora-easy-karma utilizing the
API so that when an update is pushed for a package for which test cases
have been created under this system, they will link to those test cases;
and when an update is pushed for a critical path package, they will be
able to display separately (and more prominently, perhaps) the list of
test cases relevant to the critical path functionality of the package.

Comments, suggestions and rotten fruit welcome I'm particularly
interested in feedback from package maintainers and QA contributors in
whether you feel, just after reading these pages, that you'd be
confident in going ahead and creating some test cases, or if there's
stuff that's scary or badly explained or that you feel like something is
missing and you wouldn't know where to start, etc.

The trac ticket on this is probably valuable for background, explaining
why some things in the proposal are the way they are:

https://fedorahosted.org/fedora-qa/ticket/154

it also mentions one big current omission: dependencies. For instance,
it would be very useful to be able to express 'when yum is updated, we
should also run the PackageKit test plan' (because it's possible that a
change in yum could be fine 'within itself', and all the yum test cases
pass, but could break PackageKit). That's rather complex, though,
especially with a Wiki-based system. If anyone has any bright ideas on
how to achieve this, do chip in! Thanks.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 12-21-2010, 09:53 PM
Kevin Fenzi
 
Default Package-specific test case and critical path test case project: drafts for review

On Tue, 21 Dec 2010 17:11:15 +0000
Adam Williamson <awilliam@redhat.com> wrote:

> Hi, everyone. So, in the recent debate about the update process it
> again became clear that we were lacking a good process for providing
> package-specific test instructions, and particularly specific
> instructions for testing critical path functions.

...snip...

> Comments, suggestions and rotten fruit welcome I'm particularly
> interested in feedback from package maintainers and QA contributors in
> whether you feel, just after reading these pages, that you'd be
> confident in going ahead and creating some test cases, or if there's
> stuff that's scary or badly explained or that you feel like something
> is missing and you wouldn't know where to start, etc.

...snip...

Great work Adam and QA folks.

From a quick glance over this looks good to me, and I would be happy to
start trying to make test cases. A few random things:

* Would it be worth noting that anyone can make a test case, it doesn't
need to be the maintainer, right?

* Also, might be worth noting that if you run into a specific bug as a
maintainer and fix it, thats a great time to go add a test case to
specifically test that (since it's fresh in your mind).

* finally, it's ok to just start in on some tests and add them over
time, right? We don't want to care if a package has incomplete
coverage right off the bat right?

>
> it also mentions one big current omission: dependencies. For instance,
> it would be very useful to be able to express 'when yum is updated, we
> should also run the PackageKit test plan' (because it's possible that
> a change in yum could be fine 'within itself', and all the yum test
> cases pass, but could break PackageKit). That's rather complex,
> though, especially with a Wiki-based system. If anyone has any bright
> ideas on how to achieve this, do chip in! Thanks.

Yeah, tough one. Not sure how best to handle that. Perhaps just a
'Dependencies:' type header asking you to make sure you test dependent
packages and see their test cases?

Thanks again for working on this!

kevin
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 03:05 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org