What do you think about doing the following change in
/usr/portage/profiles/targets/developer/make.defaults:

replace "test" with "test-fail-continue" to make it just less
frustrating (we still have a lot of test failures)

Hopefully that will also make more of us use the developer profile, and
detect test failures.

What do you think?

Paweł

On Fri, 04 Jun 2010 17:11:45 +0200
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:

[..]

> What do you think?

I've never felt any need or obligation to use a developer profile. I
don't think I ever saw any announcement to that effect either. What is
the use of a developer profile?[1]

Someone in the know, please sell it to me.


Regards,
jer


[1] I've seen developers complain more and more about failing test
suites. Maybe that's a related issue? Developers now use the
FEATURES set out in a developer profile and can then extract some
kind of validity claim from the fact that I obviously didn't do my QA?
That would explain a lot.

On Fri, 4 Jun 2010 18:48:38 +0200
Jeroen Roovers <jer@gentoo.org> wrote:

> [1] I've seen developers complain more and more about failing test
> suites. Maybe that's a related issue? Developers now use the
> FEATURES set out in a developer profile and can then extract some
> kind of validity claim from the fact that I obviously didn't do my QA?
> That would explain a lot.

That came out wrong.

s|from the fact|to the effect|

On Fri, 04 Jun 2010 17:11:45 +0200
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:
>
> replace "test" with "test-fail-continue" to make it just less
> frustrating (we still have a lot of test failures)
>
> Hopefully that will also make more of us use the developer profile, and
> detect test failures.
>
> What do you think?

I would say it's an improvement only because it might prevent one or two
people from completely disabling it first chance they get.

IMO, test failures should be given the same status as build failures.
Packages shouldn't be commited until they're fixed or bypassed. Following
that reasoning, FEATURES="test" is the correct setting for the dev profile.
It _should_ be annoying when you hit it, that's the point. Fix it! What's
the point of even having a test suite if it always fails? You'd be better off
to RESTRICT it and save yourself some bug reports from me and all the
other users you're foisting build errors on.

But in the real world it seems it's just never going to happen. I've been
arguing this for years but people simply don't care. It doesn't help that we
don't have any finer control than "on" or "off". I'd like to be able to say
things like "these tests should only be run by developers" or "some failures
are normal" or "hope you have 10 hours to run this" or "don't run these as
root" or "don't run tests on arm" etc etc. I'd like a pony while I'm at it.

Sorry about the rant. This is one of my biggest long-standing annoyances.

Um, so yeah. For it!


--
fonts, there's a hole in my neighbourhood
gcc-porting, down which of late i cannot help but fall
wxwidgets @ gentoo EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

Ryan Hill <dirtyepic@gentoo.org> said:
> On Fri, 04 Jun 2010 17:11:45 +0200
>
> "Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:
> > What do you think about doing the following change in
> > /usr/portage/profiles/targets/developer/make.defaults:
> >
> > replace "test" with "test-fail-continue" to make it just less
> > frustrating (we still have a lot of test failures)
> >
> > Hopefully that will also make more of us use the developer profile,
> > and detect test failures.
> >
> > What do you think?
>
> I would say it's an improvement only because it might prevent one or
> two people from completely disabling it first chance they get.
>
> IMO, test failures should be given the same status as build failures.
> Packages shouldn't be commited until they're fixed or bypassed.
> Following that reasoning, FEATURES="test" is the correct setting for
> the dev profile. It _should_ be annoying when you hit it, that's the
> point. Fix it! What's the point of even having a test suite if it
> always fails? You'd be better off to RESTRICT it and save yourself
> some bug reports from me and all the other users you're foisting build
> errors on.
>
> But in the real world it seems it's just never going to happen. I've
> been arguing this for years but people simply don't care. It doesn't
> help that we don't have any finer control than "on" or "off". I'd
> like to be able to say things like "these tests should only be run by
> developers" or "some failures are normal" or "hope you have 10 hours
> to run this" or "don't run these as root" or "don't run tests on arm"
> etc etc. I'd like a pony while I'm at it.
>
> Sorry about the rant. This is one of my biggest long-standing
> annoyances.
>
> Um, so yeah. For it!

as it seems, there is disagreement about the issue among developers.
Perhaps the council would like to settle this, so that we can go on with
our lives.

i do agree, that all packages should build successfully including the test
phase. RESTRICTing the test and an open bug when this is not the case.

thanks
Thilo

On Mon, 7 Jun 2010 12:10:04 +0200
Thilo Bangert <bangert@gentoo.org> wrote:

> i do agree, that all packages should build successfully including the
> test phase. RESTRICTing the test and an open bug when this is not the
> case.

I see more and more calls for either 1) "fixing the test suite", as if
that is suddenly not an UPSTREAM issue but the ebuilds' maintainers' or
2) setting RESTRICT=test, which would have everyone ignore the
useful aspects of a package's test suite.

The main problem with RESTRICT=test is that it's too restrictive - it
prevents a normal emerge(1) or ebuild(1) run from entering the test
phase at all, with no way to work around it, except by editing the
ebuild to remove the restriction.

======
marga /keeps/gentoo/local/app-portage/foobar # ebuild foobar-1.ebuild
test Forcing test.
* checking ebuild
checksums ;-) ... [ ok ]
* checking auxfile
checksums ;-) ... [ ok ]
* checking miscfile
checksums ;-) ... [ ok ]
* CPV: app-portage/foobar-1
* REPO: JeR
* USE: elibc_glibc kernel_linux ppc test userland_GNU
>>> Unpacking source...
>>> Source unpacked in /var/tmp/portage/app-portage/foobar-1/work
>>> Compiling source in /var/tmp/portage/app-portage/foobar-1/work ...
>>> Source compiled.
* Skipping make test/check due to ebuild restriction.
>>> Test phase [explicitly disabled]: app-portage/foobar-1
marga /keeps/gentoo/local/app-portage/foobar # cat foobar-1.ebuild
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

SLOT="0"
RESTRICT="test"
======

If you believe that test suites are a Good Thing, then you should view
RESTRICT=test as the ultimate solution to fix the problem of a broken
test suite that will never be fixed. Now, in case a test suite could
actually be destructive, dangerous to run on an unprepared system, then
there RESTRICT=test is a fine solution.

When instead a test suite should do a SKIP but erroneously does a FAIL,
then RESTRICT=test is not the solution. Fixing the test suite is, but
then that's not the maintainer's problem, but upstream's. Oddly enough
we have QA checks in place (for ICEs, for instance) that direct users
directly to upstream (through the HOMEPAGE variable), when it's
suddenly the maintainer's problem if a package fails its test suite
(because of FEATURES=userpriv or a missing kernel feature, perhaps -
nothing the maintainer can prepare the user's system for). There's
currently no way to describe the test suite's requirements to the user
except by building in many checks or perhaps by simply listing them in
an ewarn.

Since the variable will be passed on to every next ebuild you're stuck
with it, unless you are prepared to edit out RESTRICT=test, see if the
new version still fails, then edit it in again when you find nothing
has been fixed. This in turn doesn't make for easy ebuild maintenance.

There's something wrong with the way we do test phases, the way some
of us rely on them to foretell the stability or quality of a package
version, and the way we stop test suites from failing (by
only having a binary RESTRICT=test).

We could easily extend metadata.xml to describe the test phase to the
developer and user right now, for one thing, and aim for a more
automated approach to fixing the binary problem in the future.

Another solution is to change how emerge(1) and ebuild(1) deal with
RESTRICT=test. On the one hand FEATURES=test should be enabled
by testers and users with caution, as many test suites simply don't do
what you might think they do - there is no guarantee that a program
will run well in the Real World by merely satisfying its (limited?) test
conditions. With that in mind, RESTRICT=test should perhaps not block
testing the way that it does now.


jer

On 6/7/10 12:10 PM, Thilo Bangert wrote:
> as it seems, there is disagreement about the issue among developers.
> Perhaps the council would like to settle this, so that we can go on with
> our lives.
>
> i do agree, that all packages should build successfully including the test
> phase. RESTRICTing the test and an open bug when this is not the case.

Thilo, I'm trying not to touch the issue of test failures at all. Also,
my feeling is that although not too many people commented here, we have
a consensus in favor of my suggestion.

And my suggestion is to replace FEATURES="test" in the developer profile
with FEATURES="test test-fail-continue" to make us developers actually
use it.

I've been running my dev box with FEATURES="test test-fail-continue" and
I'm happy. I'm filing bugs for the test failures, but the packages get
installed regardless of that (i.e. I get the work done).

If it does answer your doubts, then great. If it doesn't, please let me
know what I'm missing.

Paweł

you make valid points regarding the overall improvement of the handling of
test suites. I am not opposed to something like that being done...

it still seems like there is agreement around the fact that something
needs to be done about src_test. currently you cant run a system which
generally enables this phase.

however, the fact that different people see different problems, should not
stop us of from solving any problem. so as a small incremental step, the
method of RESTRICTing failing tests is acceptable despite the negative
consquences you mention.

thanks

kind regards
Thilo

On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:

The following change has now landed in CVS:

Index: targets/developer/make.defaults
================================================== =================
RCS file: /var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -B -r1.3 -r1.4
--- targets/developer/make.defaults 4 Oct 2009 09:44:27 -0000 1.3
+++ targets/developer/make.defaults 9 Jun 2010 17:03:37 -0000 1.4
@@ -1,8 +1,8 @@
# Copyright 1999-2008 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
-# $Header:
/var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v 1.3
2009/10/04 09:44:27 ssuominen Exp $
+# $Header:
/var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v 1.4
2010/06/09 17:03:37 phajdan.jr Exp $

-FEATURES="collision-protect cvs digest multilib-strict sign splitdebug
stricter test userpriv usersandbox"
+FEATURES="collision-protect cvs digest multilib-strict sign splitdebug
stricter test test-fail-continue userpriv usersandbox"

# Disable branding (from desktop)
USE="-branding"

On Wed, Jun 09, 2010 at 07:08:44PM +0200, "Paweee Hajdan, Jr." wrote:
> On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
> > What do you think about doing the following change in
> > /usr/portage/profiles/targets/developer/make.defaults:
>
> The following change has now landed in CVS:

I'd suggest a dev-announce in the future on that one w/ some lead
time... dev profile is admittedly 'dev', but changes that can induce
an hour build taking a day due to tests being ran is usually good to
give a heads up on.

Aside from that, change makes sense.
~harring