About tests needing internet connection to run
 

On 8 July 2012 00:38, Pacho Ramos <pacho@gentoo.org> wrote:
> After reading:
> https://bugs.gentoo.org/show_bug.cgi?id=424719
> https://bugs.gentoo.org/show_bug.cgi?id=397973
>
> Looks like there is not consensus about how to handle this cases,
> probably a PROPERTIES variable for this would help :-/
>
> Any ideas on this kind of issue?

I've been handling it on the perl-experimental overlay by specifying
SRC_TEST="network"

SRC_TEST is a perl-module.eclass variable that controls weather or not
to run the packages inbuilt tests.

Its not anywhere in official capacity, but I think my approach is
sane-ish somewhat, just needs better native support in my opinion.

https://github.com/kentfredric/perl-experimental/blob/eclass-moretests/eclass/perl-module.eclass#L283

I think it would be nice to mask packages by their test failure
expectancy, for instance, mask packages that the tests are known to
fail on, or have tests turned on for all packages except packages
where failures are expected from tests. ( There are a few packages
which will always fail tests apparently, and it would be nice to
indicate as such in the ebuild ).

This way you can also probably opt for:
a) installing only packages which don't require network for their tests
b) only testing packages which don't require network for their tests

I've also thought it might be nice to have a way to enable testing
every time I install a ~amd64 package, instead of having a wide
spectrum "all or nothing" approach.





--
Kent

perl -e "print substr( "edrgmaM SPA NOcomil.ic@tfrken", $_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

About tests needing internet connection to run
 

On Sat, 07 Jul 2012 14:38:59 +0200
Pacho Ramos <pacho@gentoo.org> wrote:

> After reading:
> https://bugs.gentoo.org/show_bug.cgi?id=424719
> https://bugs.gentoo.org/show_bug.cgi?id=397973
>
> Looks like there is not consensus about how to handle this cases,
> probably a PROPERTIES variable for this would help :-/
>
> Any ideas on this kind of issue?

To be honest, I think the first thing to do would be fixing the test
suites to skip tests which fail due to internet connection being
unavailable. Well, there would still be question how to reliably
determine that...

--
Best regards,
Michał Górny

About tests needing internet connection to run
 

On Sat, Jul 7, 2012 at 11:21 AM, Michał Górny <mgorny@gentoo.org> wrote:
> To be honest, I think the first thing to do would be fixing the test
> suites to skip tests which fail due to internet connection being
> unavailable. Well, there would still be question how to reliably
> determine that...

For some packages, e.g. geocode-glib, which is basically a library for
calling a particular web service from C code, running the test suite
without network access is almost pointless. (Unless, of course, you
feel like implementing a clone of that web service just to run the
test suite.)

I don't like tests that need network access, but in a few cases, they
are the only to automatically verify that a package works.

-Alexandre.

About tests needing internet connection to run
 

On Sat, 7 Jul 2012 17:46:49 -0400
Alexandre Rostovtsev <tetromino@gentoo.org> wrote:

> On Sat, Jul 7, 2012 at 11:21 AM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > To be honest, I think the first thing to do would be fixing the test
> > suites to skip tests which fail due to internet connection being
> > unavailable. Well, there would still be question how to reliably
> > determine that...
>
> For some packages, e.g. geocode-glib, which is basically a library for
> calling a particular web service from C code, running the test suite
> without network access is almost pointless. (Unless, of course, you
> feel like implementing a clone of that web service just to run the
> test suite.)
>
> I don't like tests that need network access, but in a few cases, they
> are the only to automatically verify that a package works.

And 'skipped' tests simply mean that the test suite was unable to
verify whether the package works for one reason or another. Well, other
than build-time failures and a few possible runtime failures.

You just have to ensure that it correctly notices the difference
between 'no internet' and 'no matching API there'. Probably the domain
resolution failure should be the borderline.

Well, and I don't really mind having PROPERTIES about it. Some users
may actually want to know that tests could do better with internet
access.

--
Best regards,
Michał Górny