FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Packaging

 
 
LinkBack Thread Tools
 
Old 02-06-2012, 11:37 AM
Markus Mayer
 
Default Introducing a directory for installed test programs

On 02/06/2012 04:59 AM, Julio Merino wrote:

Hello,

The purpose of this long email is to justify the need for a /usr/tests
directory and to ask for approval for its creation, with the final goal
of packaging ATF, its tests and the tests provided by arbitrary
applications (e.g. the just-approved lutok package[1]). This email is
long because I feel the need to provide tons of background to explain
why this directory is required. I hope the text is clear and that it
makes enough sense to convince you of its approval. Here we go!

<snip>

The first thing coming in my mind was using something like
/usr/pkg1/test and put a configuration file for each package
(containing path and other stuff) into something like
/etc/tests/conf.d/. The benefit of this would be that additional
information can be stored for each test.


Just my 2 cents

regards

Markus

--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-06-2012, 11:46 AM
Tom Callaway
 
Default Introducing a directory for installed test programs

On 02/06/2012 01:37 PM, Markus Mayer wrote:
> The first thing coming in my mind was using something like
> /usr/pkg1/test and put a configuration file for each package
> (containing path and other stuff) into something like
> /etc/tests/conf.d/. The benefit of this would be that additional
> information can be stored for each test.

The FHS explicitly forbids that:

Large software packages must not use a direct subdirectory under the
/usr hierarchy.

~tom

==
Fedora Project
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-06-2012, 01:58 PM
Julio Merino
 
Default Introducing a directory for installed test programs

On 2/6/12 7:27 AM, Tom Callaway wrote:

On 02/06/2012 04:59 AM, Julio Merino wrote:

Hello,

[...]

Hello Tom,

Thanks for the detailed reply. A longer reply comes below


So, the first thought I had was "these binaries won't be in the $PATH".
You say above that "End users need to be encouraged to 'cd /usr/tests'
and to run stuff from there, just as they do with /usr/bin or
/usr/sbin." ... but end users do not cd into /usr/bin to run a binary
from there, they simply call the binary name and expect it to work.
I'm assuming that the fact that these tests are not in the $PATH is a
feature, not a bug.


Correct. They are not supposed to be in the PATH. (They could be
though, but their utility would be reduced and they'd just screw up
autocompletion. Also, this might not be totally doable given that
testsdir is a tree with multiple nested subdirectories; see below for a
detailed example showcasing corner cases.)


It'd be argued that the tools to run the tests (atf-run in this case)
should automatically look for stuff in /usr/tests unless told otherwise,
thus hiding the details of where the tests are stored. I'll grant you
that this is a nice idea to pursue, but that's not how things work at
the moment and I'm not sure about what the ramifications of such change
would be.


(The idea of having to cd into the tests directory before running the
tests is that you can have tests in multiple "roots". One root is the
testsdir, where the tests for all installed packages live. But other
valid roots are the top directory of your source trees, so that you can
execute the tests while doing development before doing an actual
installation!)



If this is the case, then it may be possible after all to use
/usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
area, the FHS doesn't mention it at anywhere (despite repeated pleading
to them to do so), it comes from the GNU Coding Standards and Fedora has
adopted it as an exception. The Fedora Packaging Guidelines say:

Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
as the directory for executable programs that are designed primarily
to be run by other programs rather than by users.

That said, I could definitely see the argument being made that
/usr/libexec/tests is a perfectly acceptable use case for binaries that
are "designed primarily to be run by testing frameworks, but also
available outside of $PATH for users to manually execute".


I see. /usr/libexec/tests doesn't sound too bad considering this
description! However, the data files issue you raised could get in the
way. Some more details below.



However, whether it is /usr/tests or /usr/libexec/tests, there is going
to be a need to tightly constrain the acceptable files that could go
into /usr/tests and document exactly how they are allowed to go into it.

For example, my experience with test suites is almost certainly far less
than yours, but there seems to also be a fair amount of non-executable
test data content that comes along with the executable tests. E.g.
test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
/usr/share/tests/$program/ ?


As it turns out, I forgot to mention this exact point (data files) in my
original email. The way things work at the moment within ATF is to have
data files and helper binaries in the same directory as the test program
that needs them. The test programs can then dynamically query what
their "source directory" is so that they can locate these files
regardless of where the test program lives.


The rationale for this was two-folded. First, it allows the test
programs to run both from the source directory and the installed tree
because they don't hardcode absolute paths to their helpers. And
second, the whole tests tree lives in a self-contained directory that
can either be pruned from the system (which is convenient due to the way
NetBSD works) or shared on the network. Looking at existing practice, a
test tree is composed mostly of binaries or scripts; the percentage of
data files is very low. (There was also an idea of being able to
provide such programs as "bundles" that could be copied to slave
machines for remote testing, but has never been implemented yet.)


In other words, the layout for your example would be:

testsdir/
testsdir/xpdf/
testsdir/xpdf/parser_test - Binary; atf-run executes it from within
a temporary directory to "isolate" it from the read-only testsdir
and other parts of the system. parser_test can query its "srcdir",
which in this case is "testsdir/xpdf/", and thus locate where the
00-broken.pdf file is.
testsdir/xpdf/00-broken.pdf - The auxiliary data file for the xpdf
tests.

This may rule out the suitability of /usr/libexec/tests as the location
for these files, given that your description seems to imply that libexec
is for binaries only. But if it isn't, I'd be OK with using this directory.



It is also possible that introducing a new home for binaries would cause
multi-lib heartburn in RPM, but I suspect that is a simple enough
problem to fix, since we haven't really hit it with our use of
/usr/libexec to date.


What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.



You probably should open a ticket with the FHS working group:
https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
to them so that it has a chance of being officially added in the next
major FHS revision, whenever that may be.


OK, I can do that. Wouldn't it be easier to illustrate the point /
convince them if there was existing practice somewhere? (Not sure if
BSD flavors count in this case!)



Above and beyond your proposal, the only other way I can think of
handling this would be an elaborate namespacing of the test binaries in
/usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
solution that would permit a strict compliance with the FHS as it exists
today, I'm not very excited about it and am not seriously recommending
it as an alternative.


I don't think this would be an appropriate layout, and it doesn't play
well with the structure of atf (let aside the clutter that would appear
in /usr/bin). Let me provide some additional details of how the tests
tree is structured.


Each directory in testsdir has an Atffile that indicates which test
programs exist in that directory, which subdirectories to recurse into,
and additional properties that indicate meta-data for the test programs.
So, for example, the testsdir would look like this:



testsdir/ - The top-level directory. This directory would be owned
by a filesystem package and be a dependency of anything that places
files in testsdir/.

testsdir/Atffile - The main control file. This has the knowledge to
recurse into the direct subdirectories of testsdir/ looking for
other Atffiles to execute. E.g. this would find testsdir/a/Atffile
and testsdir/b/Atffile to source tests, but it would not find
testsdir/c/Notatf nor testsdir/d/e/Atffile. This file would be
installed by the "atf" tools package, as it does not make any
sense to have it without the corresponding tools.

testsdir/lutok/ - Owned by a lutok-tests package, including all the
subfiles and subdirectories.

testsdir/lutok/Atffile - The control file for the tests provided by
lutok. Contains the list of test programs to run and any optional
meta-data for them. It also includes other subdirectories to
"recurse" into.

testsdir/lutok/operations_test - A test program for the "operations"
module of the source tree. (The name of the binary is arbitrary.)

testsdir/lutok/state_test - Another test program.

testsdir/lutok/detail/ - A subdirectory of lutok's tests containing
tests for internal modules only. (This would mimic the structure
of the source package.)

testsdir/lutok/detail/Atffile - Same as above, but for the tests in
this subdirectory.

testsdir/lutok/detail/internal_test - Yet another test program.

testsdir/lutok/detail/state_test - Another test program. Note that
this can have the same name as the program in the parent
directory! There is nothing to prevent this, and thus flattening
the test programs into /usr/bin as you propose would be very
complicated.

testsdir/lutok/foo/ - Another subdirectory for additional tests, etc.

testsdir/foobar/ - Another package, with the same structure as above.

testsdir/baz/ - Another package, but this time without any Atffile in
it. This would not be executed by atf-run because the top-level
Atffile would discard this subdirectory. (Kyua, the replacement
of atf, could run these tests though provided that a Kyuafile was
created within the subdirectory. This is out of scope now though.)


A little detail to note above is that in a project with a good amount of
test coverage, there will potentially be a test program per each source
file. This means that the testsdir contains lots of binaries, and you
don't want that many, potentially with conflicting names, anywhere in
your path! To illustrate the point, check out this output page which
contains the results of the execution of the NetBSD test suite:


http://www.whooppee.com/~paul/amd64-results/last_atf.html

Anyway. This layout may remind you a lot of make(1) and a traditional
source tree populated with Makefiles. And actually, this is the main
idea of how the tests tree is organized and is an artifact of ATF's
origins (NetBSD's source tree). However, having played with other large
source code bases, I can say that this organization is not unique to NetBSD.


We'd discuss extensively whether this layout is suboptimal for Fedora's
guidelines or not... But please note that all I'm trying to achieve
here is to figure out how to package an existing piece of software, not
how to re-architect it!


If you made it this far again, thank you.

--
Julio Merino
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-06-2012, 02:12 PM
Ralf Corsepius
 
Default Introducing a directory for installed test programs

On 02/06/2012 01:46 PM, Tom Callaway wrote:

On 02/06/2012 01:37 PM, Markus Mayer wrote:

The first thing coming in my mind was using something like
/usr/pkg1/test and put a configuration file for each package
(containing path and other stuff) into something like
/etc/tests/conf.d/. The benefit of this would be that additional
information can be stored for each test.


The FHS explicitly forbids that:

Large software packages must not use a direct subdirectory under the
/usr hierarchy.


I'd use a directory under %{libdir}/<pkg> (aka. pkglibdir).

Ralf

--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-06-2012, 02:30 PM
Tom Callaway
 
Default Introducing a directory for installed test programs

On 02/06/2012 03:58 PM, Julio Merino wrote:
> What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
> from the term and a couple of Google searches did't provide much
> insight.) FWIW, I have a preliminary atf.spec file here that provides
> subpackages for libraries, tests and binaries and things seem to be fine
> to my untrained eye. I can share this file if it helps at all.

No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).

However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/

I'm not sure how much value there is in running the test suites for
supported architectures aside from the default though. If there isn't a
desire for this, we just need to explicitly state that packages
containing test-suites cannot be multi-arch.

>> You probably should open a ticket with the FHS working group:
>> https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
>> to them so that it has a chance of being officially added in the next
>> major FHS revision, whenever that may be.
>
> OK, I can do that. Wouldn't it be easier to illustrate the point /
> convince them if there was existing practice somewhere? (Not sure if
> BSD flavors count in this case!)

Honestly, I don't know. Common sense says yes, but common sense has yet
to appear in the FHS revision process.

Fedora does not have to wait for the FHS to approve it, but we do want
the things that we adopt to also be driven through the FHS approval
process with the hopes that we can "merge" these exceptions "upstream".

>> Above and beyond your proposal, the only other way I can think of
>> handling this would be an elaborate namespacing of the test binaries in
>> /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
>> solution that would permit a strict compliance with the FHS as it exists
>> today, I'm not very excited about it and am not seriously recommending
>> it as an alternative.
>
> I don't think this would be an appropriate layout, and it doesn't play
> well with the structure of atf (let aside the clutter that would appear
> in /usr/bin). Let me provide some additional details of how the tests
> tree is structured.

[LAYOUT INFO SNIPPED]

Yes, I understand this. If we were to propose such a layout, we'd put
all the other stuff in say, /usr/share/tests/$pkg/, then provide
symlinks for the binaries into that hierarchy.

But like I said, I don't really like it.

> A little detail to note above is that in a project with a good amount of
> test coverage, there will potentially be a test program per each source
> file. This means that the testsdir contains lots of binaries, and you
> don't want that many, potentially with conflicting names, anywhere in
> your path! To illustrate the point, check out this output page which
> contains the results of the execution of the NetBSD test suite:

This further illustrates why I don't really like it. I was simply
stating it because inevitably, someone else would, and I wanted it on
the record has having been considered and discarded.

~tom

==
Fedora Project
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-07-2012, 08:36 AM
Panu Matilainen
 
Default Introducing a directory for installed test programs

On 02/06/2012 05:30 PM, Tom Callaway wrote:

On 02/06/2012 03:58 PM, Julio Merino wrote:

What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.


No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).

However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/


Um, rpm doesn't do multilib conflict resolution based on specific
directories but file type, essentially 32bit vs 64bit ELF. So if
foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
/usr/libexec/tests/foo, the file from the preferred arch (normally the
64bit one) will "win", the other file is not installed at all and no
conflict is raised.


Whether you actually *want* that behavior is another question: tests and
their associated data can just as well be arch-specific or
arch-independent. The only safe assumption is to assume arch-specific
and put all test executables and associated data into arch-specific
paths (whether some variant of lib vs lib64 style differentation or by
actually embedding the arch string).


- Panu -

--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-07-2012, 09:04 AM
Ralf Corsepius
 
Default Introducing a directory for installed test programs

On 02/07/2012 10:36 AM, Panu Matilainen wrote:

On 02/06/2012 05:30 PM, Tom Callaway wrote:

On 02/06/2012 03:58 PM, Julio Merino wrote:

What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.


No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).

However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/


Um, rpm doesn't do multilib conflict resolution based on specific
directories but file type, essentially 32bit vs 64bit ELF. So if
foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
/usr/libexec/tests/foo, the file from the preferred arch (normally the
64bit one) will "win", the other file is not installed at all and no
conflict is raised.

Whether you actually *want* that behavior is another question: tests and
their associated data can just as well be arch-specific or
arch-independent. The only safe assumption is to assume arch-specific
and put all test executables and associated data into arch-specific
paths (whether some variant of lib vs lib64 style differentation or by
actually embedding the arch string).


%{_libdir}/<package>/<somewhere> avoids all these issues.

Ralf

--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-07-2012, 09:42 AM
Panu Matilainen
 
Default Introducing a directory for installed test programs

On 02/07/2012 12:04 PM, Ralf Corsepius wrote:

On 02/07/2012 10:36 AM, Panu Matilainen wrote:

On 02/06/2012 05:30 PM, Tom Callaway wrote:

On 02/06/2012 03:58 PM, Julio Merino wrote:

What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be
fine
to my untrained eye. I can share this file if it helps at all.


No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).

However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/


Um, rpm doesn't do multilib conflict resolution based on specific
directories but file type, essentially 32bit vs 64bit ELF. So if
foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
/usr/libexec/tests/foo, the file from the preferred arch (normally the
64bit one) will "win", the other file is not installed at all and no
conflict is raised.

Whether you actually *want* that behavior is another question: tests and
their associated data can just as well be arch-specific or
arch-independent. The only safe assumption is to assume arch-specific
and put all test executables and associated data into arch-specific
paths (whether some variant of lib vs lib64 style differentation or by
actually embedding the arch string).


%{_libdir}/<package>/<somewhere> avoids all these issues.


Yes, except for noarch packages where %{_libdir} is not meaningful.

- Panu -

--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-07-2012, 10:26 AM
Michael Schwendt
 
Default Introducing a directory for installed test programs

On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:

> The purpose of this long email is to justify the need for a /usr/tests
> directory

> To put this in the context of a Linux distribution, let's consider a
> fictitious example: imagine that glib and gtk came with ATF-based tests.
> These tests would get installed into testsdir/glib/... and
> testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run"
> to execute the tests on his machine,

> 2) The second is that ATF is not encoded anywhere in the path. I.e. the
> directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
> /usr/tests/atf/pkg*. The reason for this is that the package-specific
> tests needn't be implemented using the ATF libraries.

Still, the tests must be compatible with the testing framework's set of
tools. That's enough of a reason to add some sort of namespace to the
root directory.

I also find it contradictory how you refer to "atf-run", "atf-report"
and "a collection of tools", which have "atf-" in their name. Apparently,
they are stored in global search path for executables (and therefore it
is reasonable to prefix them with something to avoid creating too generic
names, and hopefully you wouldn't propose dropping the "atf-" prefix from
their names either).

Surely the testing framework would not be renamed regularly. And
software authors, who would use this framework for writing own tests,
would need to be able to rely on an interface for retrieving filesystem
path details anyway. For example, a pkg-config file that can be queried.
Or a similar atf-config script that returns various paths.

> In fact, Kyua,
> the replacement for ATF, is able to execute these same unmodified tests
> and can also execute tests written other frameworks. In a future world,
> packages would install arbitrary tests into /usr/tests written using
> their preferred libraries (e.g. pyunit, autotest, etc.) and a single
> tool (Kyua) would run them all. Encoding the tool name in the directory
> introduces a dependency on the tool that does not necessarily exist.

Implementation details here would need to be discussed. The names of the
tools are irrelevant so far. The various testing frameworks would need to
be compatible somehow and meet the requirements of the controlling tools.
One couldn't dump arbitrary tests into a shared directory and e.g. mix
automated, semi-automated and non-automated tests.

Leaving FHS issues aside, in general it is not nice to let any package
"occupy" directories, which have very generic names, without very good
reason. That's not limited to /usr, and it's not a "first come, first served"
thing either. Imagine, a second group of testing framework authors would
also want to use a global /usr/tests in order to store incompatible tests
in there.
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-08-2012, 03:10 PM
Tom Callaway
 
Default Introducing a directory for installed test programs

On 02/07/2012 05:04 AM, Ralf Corsepius wrote:
> %{_libdir}/<package>/<somewhere> avoids all these issues.

No it doesn't. Not to mention the fact that it isn't really appropriate
to be putting binaries and architecture independent files there.

~tom

==
Fedora Project
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 

Thread Tools




All times are GMT. The time now is 10:51 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org