FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 08-27-2008, 09:10 PM
Ian Jackson
 
Default automatic bug filing by test robot

Now that gdebi-core is in back in lenny I've got my automatic package
testing robot working on lenny again. (I got stalled last time due to
leaving Canonical and being busy for a while.)

Mostly what it does right now is build a package, discover that the
package doesn't have any autopkgtest tests declared in
debian/tests/control, and then go on to the next package; it also does
test installs. This is useful as it tests buildability and
installability (but of course having functional tests in packages
would be good too - the documentation for this is currently in
/usr/share/doc/autopkgtest/README.package-tests.gz
in autopkgtest).

I'm hoping to shortly turn on the automatic bug filing mechanism.
I'm writing now to give people a chance to object :-).

Also, if you have any comments about the format of the email,
supporting references, etc., that would be helpful.

Below is an example report, which would be sent to submit@bugs
instead if I had it enabled.

Ian.

From: Ian Jackson <ian@davenant.greenend.org.uk>
To: ian@davenant.greenend.org.uk
Subject: autopkgtest lenny aolserver4: erroneous package!
Date: Mon, 25 Aug 2008 18:18:13 +0100

Source: aolserver4
Version: 4.5.0-16
User: autopkgtest@packages.debian.org
Usertags: autopkgtest

Test executed for: lenny source aolserver4
Outcome: erroneous package!
blame: arg:/volatile/ian/adt-tmp/aolserver4/src/aolserver4_4.5.0-16.dsc dsc:aolserver4
badpkg: rules build failed with exit code 2
quitting: erroneous package: rules build failed with exit code 2

erroneous package!

This message is automatically generated by the autopkgtest package
testing system. You are receiving it because:
The test results appear to indicate a problem with the package
and reports for package maintainers for lenny are being directed to
ian@davenant.greenend.org.uk

The top and tail of the test log, which is intended to be sufficient
to diagnose most failures, can be found below. However, in case this
is not sufficient, a complete log can be found along with output
files, saved temporary files, and so on, at:
http://www.chiark.greenend.org.uk/~ijackson//autopkgtest/lenny-source/

The test environment is an i386 Xen guest with limited amounts of
memory. The relevant dependencies are installed for each test but
very few other packages. Most autopkgtest reports are FTBFS problems.
Failures to satisfy build-dependencies are sometimes due to ambiguity
rather than simple lack of a suitable package.

While this bug remains open and assigned to this package, no further
retesting will be done and no additional bugs will be filed. When you
have corrected the problem, close the bug and your package will be
immediately eligible for retesting.

If this automatic testing and bug filing is undesirable for certain
packages(s) please let me know and I will add them to my blacklist.

If you have any questions about this service please contact me at:
Ian Jackson <ian@davenant.greenend.org.uk>

Regards,
Ian

-8<-
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//dists/lenny/main/source/Sources.gz ++++++
++++++ selecting ++++++
package forced: aolserver4

Package: aolserver4
Binary: aolserver4, aolserver4-core, aolserver4-dev, aolserver4-doc
Version: 4.5.0-16
Priority: optional
Section: web
Maintainer: Francesco Paolo Lovergine <frankie@debian.org>
Build-Depends: sharutils, patch, debhelper (>= 5), tcl-dev (>= 8.4), dpatch, groff, zlib1g-dev
Architecture: any
Standards-Version: 3.8.0
Format: 1.0
Directory: pool/main/a/aolserver4
Files:
61e3d55473354b30beb9db6826c9e736 1136 aolserver4_4.5.0-16.dsc
72dc769aa82aa26fb14b606bdfc523f8 3790402 aolserver4_4.5.0.orig.tar.gz
16b060f0c7ed443780476a72aa0af3cd 98812 aolserver4_4.5.0-16.diff.gz
Homepage: http://www.aolserver.com/
Checksums-Sha1:
f6617920949bd8308e0f996668a517f19d68c363 3790402 aolserver4_4.5.0.orig.tar.gz
7cde762aaf30fd6c6e33e407694b2c942ec31976 98812 aolserver4_4.5.0-16.diff.gz
Checksums-Sha256:
77f48d257e4fd73e1d9e06d42c2c64fce5a84f6b9f19709900 54734530449992 3790402 aolserver4_4.5.0.orig.tar.gz
20164d6356f2d9e099db60aa3c81620930a010e9a3dd3c2a82 b6c861d31a2b2e 98812 aolserver4_4.5.0-16.diff.gz

++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0-16.dsc ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0.orig.tar.gz ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0-16.diff.gz ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0.orig.tar.gz ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0-16.diff.gz ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0.orig.tar.gz ++++++
++++++ fetching http://mirror.relativity.greenend.org.uk/mirror/debian-ftp//pool/main/a/aolserver4/aolserver4_4.5.0-16.diff.gz ++++++
++++++ starting test ++++++
+ adt-run --tmp-dir /volatile/ian/adt-tmp/aolserver4/tmp --output-dir /volatile/ian/adt-tmp/aolserver4/out --log-file /volatile/ian/adt-tmp/aolserver4/log --summary /volatile/ian/adt-tmp/_summary --built-binaries-filter=__ --source /volatile/ian/adt-tmp/aolserver4/src/aolserver4_4.5.0-16.dsc --- adt-virt-xenlvm --userv --distro=lenny --
adt-run: trace: ** init
adt-run: trace: ** start
adt-run: trace: $ vserver: adt-virt-xenlvm --userv --distro=lenny --
adt-run: trace: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ starting
adt-run: trace: ** open, scratch=None
adt-run: trace: * initialising
adt-run: trace: * no key generation needed
adt-run: trace: * reset
adt-run: trace: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ builds ...
adt-run: trace: @@@@@@@@@@@@@@@@@@@@ dsc dsc0
adt-run: trace: * publish
adt-run: trace: $ ftparchive: sh -ec <SCRIPT> x /volatile/ian/adt-tmp/aolserver4/tmp/binaries /home/ian/.autopkgtest/gpg
adt-run: trace: $ apt-key: sh -ec <SCRIPT>
OK
Get:1 http://mirror.relativity.greenend.org.uk lenny Release.gpg [189B]
Get:2 file: Release.gpg [189B]
Get:3 http://mirror.relativity.greenend.org.uk lenny Release [74.4kB]
Get:4 file: Release [724B]
Get:5 http://mirror.relativity.greenend.org.uk lenny/main Packages/DiffIndex [2038B]
Get:6 http://mirror.relativity.greenend.org.uk lenny/contrib Packages/DiffIndex [2023B]
Hit http://mirror.relativity.greenend.org.uk lenny/non-free Packages/DiffIndex
Get:7 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-0848.13.pdiff [11.5kB]
Get:8 http://mirror.relativity.greenend.org.uk lenny/contrib 2008-08-24-2034.23.pdiff [29B]
Get:9 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-0848.13.pdiff [11.5kB]
Get:10 http://mirror.relativity.greenend.org.uk lenny/contrib 2008-08-24-2034.23.pdiff [29B]
Get:11 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-0848.13.pdiff [11.5kB]
Get:12 http://mirror.relativity.greenend.org.uk lenny/contrib 2008-08-24-2034.23.pdiff [29B]
Get:13 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-2034.23.pdiff [8602B]
Get:14 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-2034.23.pdiff [8602B]
Get:15 http://mirror.relativity.greenend.org.uk lenny/main 2008-08-24-2034.23.pdiff [8602B]
Fetched 98.9kB in 12s (7938B/s)
Reading package lists...
adt-run: trace: * publish reinstall checking...
adt-run: trace: * publish install...
adt-run: trace: * publish done
adt-run: trace: $ dsc0: python -c <SCRIPT>
adt-run: trace: $ dsc0-extract: sh -ec <SCRIPT> x /root/adt-downtmp/dsc0-build/tmpdir /root/adt-downtmp/dsc0/aolserver4_4.5.0-16.dsc
+ spec=/root/adt-downtmp/dsc0/aolserver4_4.5.0-16.dsc
++ pwd
+ origpwd=/root/adt-downtmp/dsc0-build
+ cd /root/adt-downtmp/dsc0-build
+ chown adtxenu .
+ TMPDIR=/root/adt-downtmp/dsc0-build/tmpdir
+ rm -rf -- /root/adt-downtmp/dsc0-build/tmpdir
+ export TMPDIR
+ su adtxenu -c 'mkdir -- "$TMPDIR"'
+ spec=/root/adt-downtmp/dsc0/aolserver4_4.5.0-16.dsc
+ origpwd=/root/adt-downtmp/dsc0-build
+ su adtxenu -c '
dpkg-source -x $spec
'
gpg: new configuration file `/home/adtxenu/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/adtxenu/.gnupg/gpg.conf' are not yet active during this run
gpg: Signature made Wed Jul 23 08:49:17 2008 UTC using DSA key ID E9F2C747
gpg: Can't check signature: public key not found
dpkg-source: extracting aolserver4 in aolserver4-4.5.0
dpkg-source: info: unpacking aolserver4_4.5.0.orig.tar.gz
dpkg-source: info: applying aolserver4_4.5.0-16.diff.gz
+ cd aolserver4-4.5.0/.
+ pwd
+ set +e
+ test -f debian/tests/control
+ echo 1
adt-run: trace: * <dsc:dsc0> build needed for binaries
adt-run: trace: $ dsc0: python -c <SCRIPT>
Extracting templates from packages: 20%
Extracting templates from packages: 41%
Extracting templates from packages: 61%
Extracting templates from packages: 82%
Extracting templates from packages: 100%
adt-run: trace: $ dsc0-build: sh -ec <SCRIPT> x /root/adt-downtmp/dsc0-build/tmpdir /root/adt-downtmp/dsc0-build/aolserver4-4.5.0
+ TMPDIR=/root/adt-downtmp/dsc0-build/tmpdir
+ rm -rf -- /root/adt-downtmp/dsc0-build/tmpdir
+ export TMPDIR
+ su adtxenu -c 'mkdir -- "$TMPDIR"'
+ cd /root/adt-downtmp/dsc0-build/aolserver4-4.5.0
+ dpkg-checkbuilddeps
+ su adtxenu -c 'debian/rules build'
test -d debian/patched || install -d debian/patched
dpatch apply-all
applying patch Makefile to ./ ...patch: **** Can't create file /root/adt-downtmp/dsc0-build/aolserver4-4.5.0/debian/tmp/po5iEhG8 : No such file or directory
failed.
make: *** [patch-stamp] Error 1
blame: arg:/volatile/ian/adt-tmp/aolserver4/src/aolserver4_4.5.0-16.dsc dsc:aolserver4
badpkg: rules build failed with exit code 2
adt-run: erroneous package: rules build failed with exit code 2
adt-run: trace: ** stop
adt-run: trace: ** close, scratch=tb-scratch~/root/adt-downtmp:-/|/root/adt-downtmp/!
device-mapper: remove ioctl failed: Device or resource busy
Command failed
device-mapper: remove ioctl failed: Device or resource busy
Command failed
++++++ RESULTS erroneous package! ++++++
++++++ bundling ++++++
++++++ uploading ++++++
++++++ contacting maintainer_email_override ++++++
++++++ tested. ++++++


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-27-2008, 10:20 PM
Gonéri Le Bouder
 
Default automatic bug filing by test robot

On Wed, Aug 27, 2008 at 10:10:13PM +0100, Ian Jackson wrote:
> Now that gdebi-core is in back in lenny I've got my automatic package
> testing robot working on lenny again. (I got stalled last time due to
> leaving Canonical and being busy for a while.)
>
> Mostly what it does right now is build a package, discover that the
> package doesn't have any autopkgtest tests declared in
> debian/tests/control, and then go on to the next package; it also does
> test installs. This is useful as it tests buildability and
> installability (but of course having functional tests in packages
> would be good too - the documentation for this is currently in
> /usr/share/doc/autopkgtest/README.package-tests.gz
> in autopkgtest).
>
> I'm hoping to shortly turn on the automatic bug filing mechanism.
> I'm writing now to give people a chance to object :-).
>
> Also, if you have any comments about the format of the email,
> supporting references, etc., that would be helpful.
Hi Ian,

Sorry for promoting my pet toy but I think your script would be a great input.

I'm working improving svnbuildstat to be able to support different input. The
idea is:
-a database
-a webinterface to follow the packages status
-a QA input for builds and QA checks

The current release is avalaible here: http://svnbuildstat.debian.net
It as some know limitations and bugs that I'm adressing with the new
release. This new release will:
-address package from other VCS, not only SVN,
-the database schema allow the input of an unlimited list of QA tools
(lintian, piuparts, gcc-snapshot, etc)
-and distributed buildd based on pbuilder

The development release is avalaible here: http://buildstat.debian.net.
There is still loads of things to fix but I think it'll be sane quickly
now.

There is no good documentation of the architecture yet but basicly for
a VCS it's:

VCS status:
1/ a commit is done
2/ the database is updated
3/ I try to find the tarball (e.g: ftp.debian.org or with the watchfile)
4/ I prepare a source package
5/ the database is updated

BUILDBOT:
1/ a buildbot ask for a package to build
2/ I look for the oldest package source that is not built yet
3/ I give the .dsc URL to the buildbot
4/ I do the job and upload back the result
5/ the database is updated
The idea here, is to provide the buildbot as a Debian package for Sid
users.

QA TOOL:
1/ a qa tool asks for a jobs
2/ depending on what it wants it get a source or build result to
download
3/ it checks the package
4/ it uploads the result
5/ the database is updated
the upload result is one or more string in the lintian format + a log

I also think about checking packages from the Debian repository too. I
don't see any major limitation even if it's not in my current TODO list.
I still need some weeks to get it usuable but I'm working on this
release for too long now and I really want to make it usuable for all.

For the moment:
-The DB schema is ok, an outdated picture is avalaible here:
http://nana.rulezlan.org/~goneri/svnbuildstat_devel_db_schema.png
-the buildbot script works fine
-most of the interface parts are ok, still some changes are needed
-git input script is done at 50%. The svn one is ok

Cheers,

Gonéri Le Bouder
 
Old 08-27-2008, 11:19 PM
Ian Jackson
 
Default automatic bug filing by test robot

Gonéri Le Bouder writes ("Re: automatic bug filing by test robot"):
> Sorry for promoting my pet toy but I think your script would be a
> great input.

Right ...

> I'm working improving svnbuildstat to be able to support different input. The
> idea is:
> -a database
> -a webinterface to follow the packages status
> -a QA input for builds and QA checks

Right. Note that most of the effort in my system is in
* a quickly-resettable VM containing a pretty much normal
system (although a small one) rather than a chroot
* machinery for running tests which are declared by a package
- but note that almost no packages currently declare any
tests so this is of limited use right now

The tests my framework does when a package doesn't declare any
specific function tests are mostly a smallish subset of piuparts.

> 3/ it checks the package

It would certainly be good from my point of view to have a run of the
autopkgtest package-declared functional tests here.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-28-2008, 07:19 AM
Raphael Hertzog
 
Default automatic bug filing by test robot

On Wed, 27 Aug 2008, Ian Jackson wrote:
> I'm hoping to shortly turn on the automatic bug filing mechanism.
> I'm writing now to give people a chance to object :-).
[...]
> very few other packages. Most autopkgtest reports are FTBFS problems.

I object filing FTBFS automatically. In some cases, the FTBFS is a failure
from one of the build-dependencies and filing bugs automatically would
lead to lots of useless/duplicate bugs. Thus a I think that a human filter
here is most welcome.

Lucas is covering this quite well up to now. If we ever get back to the
situation where nobody is willing to do that work, then we can reconsider
this. Also, how would you check that a bug has not already been filed
manually by another user?

Note however, I have no problem with a bot filing bugs based on functional
tests that people might have added to their packages.

Cheers,
--
Raphaël Hertzog

Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-28-2008, 08:48 AM
Martin Michlmayr
 
Default automatic bug filing by test robot

* Raphael Hertzog <hertzog@debian.org> [2008-08-28 09:19]:
> I object filing FTBFS automatically. In some cases, the FTBFS is a failure
...
> Also, how would you check that a bug has not already been filed
> manually by another user?

Totally agreed.
--
Martin Michlmayr
http://www.cyrius.com/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-28-2008, 11:19 AM
Holger Levsen
 
Default automatic bug filing by test robot

Hi Ian,

nice work! but...

On Wednesday 27 August 2008 23:10, Ian Jackson wrote:
> I'm hoping to shortly turn on the automatic bug filing mechanism.
> I'm writing now to give people a chance to object :-).

/me objects / suggests to first enable mails to some mailinglist (this one?),
instead of actually filing bugs. Then we can review the reports and then
decide whether it's working as well as it should.


regards,
Holger
 
Old 08-28-2008, 11:38 AM
Brian May
 
Default automatic bug filing by test robot

Holger Levsen wrote:
/me objects / suggests to first enable mails to some mailinglist (this one?),
instead of actually filing bugs. Then we can review the reports and then
decide whether it's working as well as it should.


Maybe send the report to <packagename>@package.debian.org instead?

As a maintainer I would like to know when a package fails to build for
any reason, even if it is due to factors outside my control.


Brian May


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-28-2008, 01:11 PM
Jonathan Wiltshire
 
Default automatic bug filing by test robot

On Thu, Aug 28, 2008 at 09:38:54PM +1000, Brian May wrote:
> Maybe send the report to <packagename>@package.debian.org instead?
>
> As a maintainer I would like to know when a package fails to build for
> any reason, even if it is due to factors outside my control.
>

I'm inclined to agree. I would find it very useful to get reports, but
not as a bug against my package.

--
Jonathan Wiltshire
 
Old 08-28-2008, 06:11 PM
Ian Jackson
 
Default automatic bug filing by test robot

Jonathan Wiltshire writes ("Re: automatic bug filing by test robot"):
> I'm inclined to agree. I would find it very useful to get reports, but
> not as a bug against my package.

That seems to be the consensus. Fair enough. I shall see about
getting a PTS tag for these reports. I'll leave them formatted as a
BTS submission so that they can be simply bounced to the BTS if
desired.

Raphael Hertzog writes ("Re: automatic bug filing by test robot"):
> Note however, I have no problem with a bot filing bugs based on functional
> tests that people might have added to their packages.

I'll have to implement this distinction but it won't be hard.

Thanks to everyone who commented.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-28-2008, 07:32 PM
Gonéri Le Bouder
 
Default automatic bug filing by test robot

On Thu, Aug 28, 2008 at 07:11:51PM +0100, Ian Jackson wrote:
> Jonathan Wiltshire writes ("Re: automatic bug filing by test robot"):
> > I'm inclined to agree. I would find it very useful to get reports, but
> > not as a bug against my package.
>
> That seems to be the consensus. Fair enough. I shall see about
> getting a PTS tag for these reports. I'll leave them formatted as a
> BTS submission so that they can be simply bounced to the BTS if
> desired.
What about buildstat.debian.net as a central place for such tests? By
providing a common QA workflow, this will help ones to had there own
QA tool.

I attached a script I use to run lintian to show how simple it can be to
add new check. You can run it by grabbing buildstat-job-manager here:
http://svn.debian.org/wsvn/collab-qa/buildstat/trunk/buildstat-job-manager/?rev=0&sc=0
svn://svn.debian.org/svn/collab-qa/buildstat/trunk/buildstat-job-manager

Lintian is just an example of a QA tool here.

Regards,

Gonéri Le Bouder
 

Thread Tools




All times are GMT. The time now is 06:59 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org