(this isn't directed at any one person or group or any recent incident, this
has been bugging me for years)
I have one simple request. When you make a non-trivial change to an ebuild -
a patch, a version bump, anything that can effect the behaviour of the
package - please run the test suite. If it fails, fix it. Or restrict it.
Or even make it non-fatal if there's no other choice. If you can reproduce
failures outside of portage, report them upstream. Failures indicate either
a broken package or a broken test suite and either way it's in your best
interests to get them fixed.
Remember that for anyone running FEATURES=test a failure breaks the build*.
You wouldn't commit something that doesn't compile (hopefully :P), so why
is this any different? There is no point in even having test suites if
everyone just disables them in frustration because every third package fails.
I apologize for the rant, but when I do testing for gcc-porting I rely
heavily on tests to catch runtime issues. And every release cycle I end up
spending way too much time trying to figure out why a test is failing, only
to find that there's been an bug open about it for two years with no
activity.
* I know about test-fail-continue, but I've found that it just causes me
file fewer bug reports because they don't annoy me as much.
--
fonts, by design, by neglect
gcc-porting, for a fact or just for effect
wxwidgets @ gentoo EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662
02-21-2010, 08:11 AM
"Paweł Hajdan, Jr."
The importance of test suites
On 2/21/10 5:08 AM, Ryan Hill wrote:
> I have one simple request. When you make a non-trivial change to an ebuild -
> a patch, a version bump, anything that can effect the behaviour of the
> package - please run the test suite.
Yeah, on my dev box I just run with FEATURES="test" all the time. Then
it's impossible to forget it. And I also catch failures in other packages.
Maybe it should get more exposure in the developer docs?
> If it fails, fix it. Or restrict it. Or even make it non-fatal if there's
> no other choice.
It's really frustrating when there is a bug reported about package
failing tests and everybody can reproduce it, yet the maintainer doesn't
at least put RESTRICT="test" in the ebuild.
Is it acceptable for another dev to jump in and add RESTRICT="test" to
an ebuild if the maintainer does not respond to a bug report in a timely
manner?
The concern here may be that it's papering over the real problem, but
the good side is that it'd make running with FEATURES=test much easier.
By the way, for packages that fail the test suite and refuse to disable
it, I change the env locally so that FEATURES=-test (via /etc/portage/env).
Paweł Hajdan jr
02-21-2010, 08:40 AM
Thilo Bangert
The importance of test suites
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> said:
> On 2/21/10 5:08 AM, Ryan Hill wrote:
> > I have one simple request. When you make a non-trivial change to an
> > ebuild - a patch, a version bump, anything that can effect the
> > behaviour of the package - please run the test suite.
>
> Yeah, on my dev box I just run with FEATURES="test" all the time. Then
> it's impossible to forget it. And I also catch failures in other
> packages.
>
> Maybe it should get more exposure in the developer docs?
>
> > If it fails, fix it. Or restrict it. Or even make it non-fatal if
> > there's no other choice.
>
> It's really frustrating when there is a bug reported about package
> failing tests and everybody can reproduce it, yet the maintainer
> doesn't at least put RESTRICT="test" in the ebuild.
>
> Is it acceptable for another dev to jump in and add RESTRICT="test" to
> an ebuild if the maintainer does not respond to a bug report in a
> timely manner?
IMHO yes! of course, the bug has to be left open until the issue is fixed
for real.
>
> The concern here may be that it's papering over the real problem, but
> the good side is that it'd make running with FEATURES=test much easier.
which is a good thing, since more tests will be run. RESTRICT="test"
should always generate a repoman QA warning IMHO.
>
> By the way, for packages that fail the test suite and refuse to disable
> it, I change the env locally so that FEATURES=-test (via
> /etc/portage/env).
how many packages do you have in there? i usually just do it manually, so
i dont have easily available stats on how many packages are affected.
>
> Paweł Hajdan jr
02-21-2010, 08:53 AM
"Paweł Hajdan, Jr."
The importance of test suites
On 2/21/10 10:40 AM, Thilo Bangert wrote:
> "Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> said:
>> The concern here may be that it's papering over the real problem, but
>> the good side is that it'd make running with FEATURES=test much easier.
>
> which is a good thing, since more tests will be run. RESTRICT="test"
> should always generate a repoman QA warning IMHO.
+1
>> By the way, for packages that fail the test suite and refuse to disable
>> it, I change the env locally so that FEATURES=-test (via
>> /etc/portage/env).
>
> how many packages do you have in there? i usually just do it manually, so
> i dont have easily available stats on how many packages are affected.
On Sat, 20 Feb 2010, Ryan Hill wrote:
[... "Please use the test suites, you're making lives easier." ...]
Also, if the test failure is "portable", you don't waste the time
of N arch maintainers that run into the same problem on waaaay
slower machines than yours.
Thanks,
Tobias
02-21-2010, 04:06 PM
Petteri Räty
The importance of test suites
On 21.2.2010 1.11, "Paweł Hajdan, Jr." wrote:
>
> Is it acceptable for another dev to jump in and add RESTRICT="test" to
> an ebuild if the maintainer does not respond to a bug report in a timely
> manner?
>
Preference order:
1. Fix the tests
2. Disable just the failing test
3. RESTRICT="test"
Regards,
Petteri
02-22-2010, 11:23 AM
"Paweł Hajdan, Jr."
The importance of test suites
On 2/21/10 10:11 AM, "Paweł Hajdan, Jr." wrote:
> On 2/21/10 5:08 AM, Ryan Hill wrote:
>> I have one simple request. When you make a non-trivial change to an ebuild -
>> a patch, a version bump, anything that can effect the behaviour of the
>> package - please run the test suite.
>
> Yeah, on my dev box I just run with FEATURES="test" all the time. Then
> it's impossible to forget it. And I also catch failures in other packages.
I just noticed that FEATURES="test" is enabled by default in the
developer profile, with some other features that make portage more strict.
It also looks like the developer profile is not mentioned in the
developer docs.
What would you think about better advocating usage of the developer
profile among developers? Looks like many problems would disappear
simply because the maintainer of a given package would hit the problem
first (and fix it before it gets into portage).
libtool was fixed by diego a couple of weeks ago, it's probably not
needed anymore.
--
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo
Mon Feb 22 15:30:01 2010
Return-path: <bounce-debian-user=tom=linux-archive.org@lists.debian.org>
Envelope-to: tom@linux-archive.org
Delivery-date: Mon, 22 Feb 2010 14:32:32 +0200
Received: from liszt.debian.org ([82.195.75.100]:43967)
by s2.java-tips.org with esmtps (TLSv1:AES256-SHA:256)
(Exim 4.69)
(envelope-from <bounce-debian-user=tom=linux-archive.org@lists.debian.org>)
id 1NjXSh-0006Az-Nl
for tom@linux-archive.org; Mon, 22 Feb 2010 14:32:32 +0200
Received: from localhost (localhost [127.0.0.1])
by liszt.debian.org (Postfix) with QMQP
id 422C413A5327; Mon, 22 Feb 2010 13:01:30 +0000 (UTC)
Old-Return-Path: <gpall@ccf.auth.gr>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on liszt.debian.org
X-Spam-Level:
X-Spam-Status: No, score=-11.0 required=4.0 tests=LDOSUBSCRIBER,LDO_WHITELIST
autolearn� version=3.2.5
X-Original-To: lists-debian-user@liszt.debian.org
Delivered-To: lists-debian-user@liszt.debian.org
Received: from localhost (localhost [127.0.0.1])
by liszt.debian.org (Postfix) with ESMTP id 829AF13A52F9
for <lists-debian-user@liszt.debian.org>; Mon, 22 Feb 2010 13:01:23 +0000 (UTC)
X-Virus-Scanned: at lists.debian.org with policy bank en-ht
X-Amavis-Spam-Status: No, score=-7 tagged_above=-10000 required=5.3
tests=[BAYES_00=-2, LDO_WHITELIST=-5] autolearn=ham
Received: from liszt.debian.org ([127.0.0.1])
by localhost (lists.debian.org [127.0.0.1]) (amavisd-new, port 2525)
with ESMTP id YjaqtnlGOyVs for <lists-debian-user@liszt.debian.org>;
Mon, 22 Feb 2010 13:01:16 +0000 (UTC)
X-policyd-weight: using cached result; rate: -7
Received: from hermes3.ccf.auth.gr (hermes3.ccf.auth.gr [155.207.1.69])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(Client CN "AUTH Mail Servers", Issuer "AUTH Network Operations Center Certification Authority R3" (not verified))
by liszt.debian.org (Postfix) with ESMTPS id 191F513A51A2
for <debian-user@lists.debian.org>; Mon, 22 Feb 2010 13:01:09 +0000 (UTC)
Received: from [155.207.112.12] (aris.ccf2.auth.gr [155.207.112.12])
(authenticated bits=0)
by hermes3.ccf.auth.gr (8.14.3/8.14.3) with ESMTP id o1MD15Hb017902
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits%6 verify=NOT)
for <debian-user@lists.debian.org>; Mon, 22 Feb 2010 15:01:06 +0200
Message-ID: <4B828011.6070805@ccf.auth.gr>
Date: Mon, 22 Feb 2010 15:01:05 +0200
From: =?UTF-8?B?zpPOuc+Oz4HOs86/z4IgzqDOrM67zrvOsc+C? <gpall@ccf.auth.gr>
User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706)
MIME-Version: 1.0
To: debian Users ENG <debian-user@lists.debian.org>
Subject: how to convince that debian is one the three major choices for a
stable server environment?
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms030600020002060303020103"
X-Virus-Scanned: clamav-milter 0.95.3 at hermes1
X-Virus-Status: Clean
X-Rc-Virus: 2007-09-13_01
X-Rc-Spam: 2008-11-04_01
Resent-Message-ID: <N5Tqna42U9K.A.ss.qAogLB@liszt>
Resent-From: debian-user@lists.debian.org
X-Mailing-List: <debian-user@lists.debian.org> archive/latest/569954
X-Loop: debian-user@lists.debian.org
List-Id: <debian-user.lists.debian.org>
List-Post: <mailto:debian-user@lists.debian.org>
List-Help: <mailto:debian-user-request@lists.debian.org?subject=help>
List-Subscribe: <mailto:debian-user-request@lists.debian.org?subject=subscribe>
List-Unsubscribe: <mailto:debian-user-request@lists.debian.org?subject=unsubscribe>
Precedence: list
Resent-Sender: debian-user-request@lists.debian.org
Resent-Date: Mon, 22 Feb 2010 13:01:30 +0000 (UTC)
This is a cryptographically signed message in MIME format.
So, yes, we are moving on from our 10year experience with gentoo, and
are searching for our new environment. From my personal experience I
would say debian stable - any hard evidence to support the claim? Server
OS statistics? Statistics for stableness? Bugs? Any white papers showing
debian's superiority?
I am also doing my google research, but I'm asking if someone can point
me to something like real hard evidence...
--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/4B828011.6070805@ccf.auth.gr
Still fails for me (https://bugs.gentoo.org/245103). Please note I'm
running x86, so this is with dev-python/pygtk-2.14.1-r1 and not the
latest version).
>> |-- dev-scheme
>> | `-- guile.env
>> `-- sys-devel
>> |-- binutils.env
>> `-- libtool.env
>
> libtool was fixed by diego a couple of weeks ago, it's probably not
> needed anymore.
This still fails for me on x86 stable, https://bugs.gentoo.org/293758.
Paweł Hajdan jr
02-22-2010, 09:23 PM
Ryan Hill
The importance of test suites
On Sun, 21 Feb 2010 10:11:25 +0100
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:
> On 2/21/10 5:08 AM, Ryan Hill wrote:
> > I have one simple request. When you make a non-trivial change to an ebuild -
> > a patch, a version bump, anything that can effect the behaviour of the
> > package - please run the test suite.
>
> Yeah, on my dev box I just run with FEATURES="test" all the time. Then
> it's impossible to forget it. And I also catch failures in other packages.
>
> Maybe it should get more exposure in the developer docs?
It's enabled by default in the developer profiles, so either devs aren't using
them or they're explicitly disabling it.
> By the way, for packages that fail the test suite and refuse to disable
> it, I change the env locally so that FEATURES=-test (via /etc/portage/env).
I used to do exactly that but it has a big disadvantage. There are
particular packages where I want to disable the test USE flag simply because
it pulls in a ton of unwanted dependencies. This can't be done with env
files; it's force {en,dis}abled by the global FEATURES setting (even
package.use doesn't override it).
Luckily, you can kill two birds with one stone by using a little trick buried
deep in the make.conf manpage: masking the test USE flag for a particular
package disables the test suite /even if that package doesn't have a test USE
flag/.
$ head -n5 /etc/portage/profile/package.use.mask
dev-db/virtuoso-server test
dev-java/commons-cli test
dev-libs/boost test
dev-libs/ppl test
dev-util/subversion test
--
fonts, by design, by neglect
gcc-porting, for a fact or just for effect
wxwidgets @ gentoo EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662