Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   CentOS Development (http://www.linux-archive.org/centos-development/)
-   -   a post-build test harness (http://www.linux-archive.org/centos-development/200388-post-build-test-harness.html)

Jeff Johnson 11-26-2008 07:18 PM

a post-build test harness
 
On Nov 26, 2008, at 12:45 PM, Karanbir Singh wrote:


Hi,

One of the things that I'd like to get working sometime is a proper
post-build test harness for rpm's. At the moment, what we do is
quite basic, along with rpmdiff and some comparison tools we do a
string of bash-written script tests. There must be another way of
doing this, a sort of rpm::unit, for the lack of a better name, but
modelled on Rspec. To illustrate :




What is "Rspec"? I'm unfamiliar with the reference.


For Pkg in PackageList: {
ensure Pkg.SIGGPG => present;
};



Makes sure that Packages we are testing are signed, I guess it could
also be ensure=> 'blahblah'; to make sure its signed with a specific
key.


A more complex test might be to make sure that content in multilib
pkgs match for overlapping content + multilib acceptable content.


An even more complex test might be to make sure any rpm that drops
an init script also has a chkconfig stanza in the %post / %pre ( as
an example ).


The first step would, of course, be to look at doing this per
package, and then growing it to work with multiple packages ( eg.
doing things like :




Per-pkg unit tests are likely best done by running rpmlint with a
custom configuration.


Multi-pkg tests require collecting an approximation to "everything"
that is
currently in the distro. How the collection is done, and whether the
picture
of "everything" is accurate, is (or has been) what has prevented
better distro

QA imho.

TestPackage: { ensure latest in -> http://mirror.centos.org/centos/updates/i386/
} ; to make sure that this new package does indeed provide an
higher EVR than whatever is in the repo at the end of that url.




That introduces a 3rd layer, in addition to unit & integration
testing, checking

that indeed, a package appears in distribution URI's as intended.

One way to avoid the test for higher is to automate R++ when building.
The problem these days is how to recognize templates for Release: so
that an autoincrement might be performed reliably. I suspect that
EVR autoincrement is an easier problem to solve than detecting
failure after the build.


And having this harness pass or fail depending on the test outcome.



The "fail" case is often painful, requiring human intervention.
A more automated "fail" proof process flow is perhaps a better
approach than a "fail" test.

OTOH, detection is better than not testing at all, even if automated
correction

or prevention (through processes) is better than "fail" detection.

Perhaps this is not the right list, but most of the people who would
end up using this or having to live with me forcing them to use it
are on this list... so I thought this would be a good place to start.


Comments ? Am I wasting my time ?



Doesn't sound like a waste of time to me.

73 de Jeff

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel


_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel

Karanbir Singh 11-28-2008 02:56 PM

a post-build test harness
 
Jeff Johnson wrote:

What is "Rspec"? I'm unfamiliar with the reference.


Essentially a unit testing framework that works on specifications rather
than tests. Let me see if I can come up with a semi functional example
to demonstrate what I have in mind.


Per-pkg unit tests are likely best done by running rpmlint with a custom
configuration.


I've looked at rpmlint a few times, let me look again. Perhaps most of
this might be plumbable via there.


That introduces a 3rd layer, in addition to unit & integration testing,
checking

that indeed, a package appears in distribution URI's as intended.


I think some / most of this info might be extractable from createrep'd
metadata. Will need to dig more into that, at the moment just looking at
individual package testing, and some collection stuff. Eg. making sure
we get the right multilib stuff into the right place.



One way to avoid the test for higher is to automate R++ when building.
The problem these days is how to recognize templates for Release: so
that an autoincrement might be performed reliably. I suspect that
EVR autoincrement is an easier problem to solve than detecting
failure after the build.


I agree with the auto bump, or check at the buildsystem level to make
sure that package-to-build is indeed higher. Its whats in place at the
moment here. However, its not easy to enforce on the main distro level
stuff, since we need to 'inhert' Rel: Ver: from upstream and match it
exactly1[]



And having this harness pass or fail depending on the test outcome.



The "fail" case is often painful, requiring human intervention.
A more automated "fail" proof process flow is perhaps a better
approach than a "fail" test.

OTOH, detection is better than not testing at all, even if automated
correction

or prevention (through processes) is better than "fail" detection.


I think you put it better than I was able to, the whole aim is to catch
these things before it breaks or causes issues on production machines
out there. We dont really churn out that many packages per day that
would make such reporting get noisy and irritating, as yet.



- KB

[1]: except packages we change, those get a .centos in the Release tag
and we can control it using homebrew scripting.


_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel

Jeff Johnson 11-28-2008 03:52 PM

a post-build test harness
 
On Nov 28, 2008, at 10:56 AM, Karanbir Singh wrote:


Jeff Johnson wrote:

What is "Rspec"? I'm unfamiliar with the reference.


Essentially a unit testing framework that works on specifications
rather than tests. Let me see if I can come up with a semi
functional example to demonstrate what I have in mind.




Yes, I found Rspec soon after I asked.

The Rspec agile approach, populating the desired test framework at
high level *even if the tests do nothing*
is dead-on imho. What tends to happen with testing is that interest
wanes, and round 'tuits go missing, so
tests that are incrementally added to a framework often don't
accomplish what was originally intended (in my experience).


Per-pkg unit tests are likely best done by running rpmlint with a
custom configuration.


I've looked at rpmlint a few times, let me look again. Perhaps most
of this might be plumbable via there.




The trick with rpmlint, like all lints, is to actively disable the
spewage until the fire hose
is merely a trickle. Then proceed by turning on whatever looks useful.
If you expect rpmlint
*by default* to find all the errors, well, your definition of "error"
is likely going to

be different thatn rpmlint's.

I watched RedHat QA struggle with Mandrake packaging policies applied
to a RedHat distro
for a year when rpmlint was introduced. Somehow the QA trolls never
could grasp the insanity
of Mandrake != RedHat packaging policies, and so using rpmlint was
eventually abandoned.


An agile Rspec approach enables the QA trolls to actively control the
testing goal rather

than reactively trying to assign a priority to various flaws.

That introduces a 3rd layer, in addition to unit & integration
testing, checking

that indeed, a package appears in distribution URI's as intended.


I think some / most of this info might be extractable from
createrep'd metadata. Will need to dig more into that, at the moment
just looking at individual package testing, and some collection
stuff. Eg. making sure we get the right multilib stuff into the
right place.




Some of the tests you wish cannot be performed by looking at single
packages, only by looking at the collection of package metadata.

The metadata aggregation can be done by createrepo, but the XML
generated by createrepo discards
information from *.rpm packages, and adds additional information from
comps.


Whether that is a good thing (or not) depends on one's testing goals.
E.g. I'm
not all sure why half of the items on the proposed verifytree feature
bullet list are

there. Obsoletes: loops? Self-referential dependencies? Whatever ...

An alternative to repo metadata might be just to import *complete*
metadata
into your own database. A database, not XML, is more useful for QA
imho. See
sophie.zarb.org for a very intelligent postgres database that has
*complete*
*.rpm metadata, where queries to index additional tag data can be
retrofitted, and

even remote queries or web display, can all be rather easily done.

All you can do with XML is download the next generation and scratch your
cornea all over again with the pointy angle brackets. ;-)

One way to avoid the test for higher is to automate R++ when
building.

The problem these days is how to recognize templates for Release: so
that an autoincrement might be performed reliably. I suspect that
EVR autoincrement is an easier problem to solve than detecting
failure after the build.


I agree with the auto bump, or check at the buildsystem level to
make sure that package-to-build is indeed higher. Its whats in place
at the moment here. However, its not easy to enforce on the main
distro level stuff, since we need to 'inhert' Rel: Ver: from
upstream and match it exactly1[]




Sure its easy to enforce, what sort of clue bat do you want added to
rpmlib? ;-)


Why do you want *exactly* the same Release:? That's much like RedHat
QA trolls

being cornfused by Mandrake packaging policies imho. If all the patches
are included, and the buildsys is configured identically for a CentOS
rebuild,
does it really matter what value is in the Release: tag? And if it
*does* matter
to CentOS, then adding a substitute for buildrelease tracking is
likely necessary.


I f the package is rebuilt, then the Release: should be bumped alwaysl
even if
nothing in the package changed, the build system and build tools are
different,
and its really hard to find issues with two apparently identical
packages that

behave differently.


And having this harness pass or fail depending on the test outcome.


The "fail" case is often painful, requiring human intervention.
A more automated "fail" proof process flow is perhaps a better
approach than a "fail" test.
OTOH, detection is better than not testing at all, even if
automated correction

or prevention (through processes) is better than "fail" detection.


I think you put it better than I was able to, the whole aim is to
catch these things before it breaks or causes issues on production
machines out there. We dont really churn out that many packages per
day that would make such reporting get noisy and irritating, as yet.




Could you post a pointer to your current CentOS QA scripts please? I've
likely asked before, but have lost the memories.

73 de Jeff
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel


All times are GMT. The time now is 04:43 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.