Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian Development (http://www.linux-archive.org/debian-development/)
-   -   Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild (http://www.linux-archive.org/debian-development/489426-wanted-system-volunteer-do-whole-archive-rebuild-3-test-sbuild.html)

Roger Leigh 02-14-2011 10:36 PM

Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild
 
Hi folks,

sbuild, which does the job of building binary packages on our buildds,
uses a built-in build-dependency resolver ("internal") to work out
what packages need installing/removing in order to satisfy a package's
Build-Depends and Build-Conflicts. Unfortunately, this has a number
of bugs, e.g. #403246, as well as serious maintainability issues which
make it less than desirable. Last year, we introduced two new
resolvers, "apt" and "aptitude" which delegate the somewhat complex
build dependency resolving to the tools which are best at it.

Now that squeeze is out, it would be great if we could revisit the
discussion about which build dependency resolver we want to use on
our buildds. The main concern here is that switching resolvers will
change exactly which packages are installed in some situations, mainly
in the case of packages depending on alternatives, which could lead to
different packages being installed, inconsistent selection of packages
being installed, or broken package builds. The intenal resolver was
dumb: it just always chose the first dependency even if it was
unsatisfiable. aptitude is far cleverer; apt somewhere in between,
though we've tweaked aptitude to behave as close to stupid as we can.

However, in order to understand the implications, we need concrete
data about exactly how they differ, and under what situations. What
I would like to do is a rebuild of the squeeze archive (since it
won't change between builds) using each of the "internal", "apt" and
"aptitude" resolvers. Since sbuild records the complete set of
packages available immediate before the build starts, we can extract
this from the build logs, find any differences, and determine what
causes them, if and when they occur.

What is needed:
- a stable/testing/unstable system
- with a lot of disc space
- and a lot of spare CPU cycles
- sbuild 0.60.9-1 (to be uploaded soon; or pull from git)
[with automatic update/upgrade disabled, and logging to alt log dir
for each run]
- schroot
- LVM
- The builds will use a cleanly debootstrapped squeeze on an LVM LV,
which can be freshly snapshotted for each build, as on buildds.
This will make the comparisons between resolvers identical.

We don't need to store the built binaries, so they could be thrown; we
just need to store the build logs. It might also be possible to skip
some of the archive; we only really need a representative sample to
do a realistic test, if time and disc space are issues.

If there are any project machines available that could do this, that
would be great!


Many thanks,
Roger

--
.'`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.

Raphael Geissert 02-15-2011 01:53 AM

Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild
 
Roger Leigh wrote:
> However, in order to understand the implications, we need concrete
> data about exactly how they differ, and under what situations. What
> I would like to do is a rebuild of the squeeze archive (since it
> won't change between builds) using each of the "internal", "apt" and
> "aptitude" resolvers. Since sbuild records the complete set of
> packages available immediate before the build starts, we can extract
> this from the build logs, find any differences, and determine what
> causes them, if and when they occur.

Am I missing something or your really don't need none of the following:

> What is needed:
[...]
> - with a lot of disc space
> - and a lot of spare CPU cycles
> - schroot
> - LVM
> - The builds will use a cleanly debootstrapped squeeze on an LVM LV,
> which can be freshly snapshotted for each build, as on buildds.
> This will make the comparisons between resolvers identical.

?
All you seem to want is the resolver's results, nothing else.
In fact, a simple, clean, chroot running apt-get --dry-run build-dep, and
aptitude --simulate build-dep should do it.

Cheers,
--
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: ijcpqp$g0f$1@dough.gmane.org">http://lists.debian.org/ijcpqp$g0f$1@dough.gmane.org

Roger Leigh 02-15-2011 09:14 AM

Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild
 
On Mon, Feb 14, 2011 at 08:53:13PM -0600, Raphael Geissert wrote:
> Roger Leigh wrote:
> > However, in order to understand the implications, we need concrete
> > data about exactly how they differ, and under what situations. What
> > I would like to do is a rebuild of the squeeze archive (since it
> > won't change between builds) using each of the "internal", "apt" and
> > "aptitude" resolvers. Since sbuild records the complete set of
> > packages available immediate before the build starts, we can extract
> > this from the build logs, find any differences, and determine what
> > causes them, if and when they occur.
>
> Am I missing something or your really don't need none of the following:
>
> > What is needed:
> [...]
> > - with a lot of disc space
> > - and a lot of spare CPU cycles
> > - schroot
> > - LVM
> > - The builds will use a cleanly debootstrapped squeeze on an LVM LV,
> > which can be freshly snapshotted for each build, as on buildds.
> > This will make the comparisons between resolvers identical.
>
> ?
> All you seem to want is the resolver's results, nothing else.
> In fact, a simple, clean, chroot running apt-get --dry-run build-dep, and
> aptitude --simulate build-dep should do it.

I don't just want to test the resolver, which I agree wouldn't need
this. I also want to know the number of build failures each resolver
results in. While this would show up from an altered package list, an
altered package list might not mean anything bad at all in most cases,
the discrepancies due to using a different resolving algorithm, which
would not alter the correctness of the outcome. Having the complete
build logs means we can then examine them in detail to look at exactly
what went wrong.


Regards,
Roger

--
.'`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.

Tollef Fog Heen 02-15-2011 09:24 AM

Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild
 
]] Roger Leigh

| If there are any project machines available that could do this, that
| would be great!

I have a couple of machines that I'd like to get burned in properly.
They're not .d.o machines (they're freedesktop.org), but I'll happily
give you access to play around. Please grab me on IRC or off-list and
we'll work something out.

--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87oc6dd28d.fsf@qurzaw.varnish-software.com">http://lists.debian.org/87oc6dd28d.fsf@qurzaw.varnish-software.com

Lucas Nussbaum 02-15-2011 10:15 AM

Wanted: system/volunteer to do whole-archive rebuild (3) to test sbuild
 
On 15/02/11 at 11:35 +0100, Marco d'Itri wrote:
> On Feb 15, Roger Leigh <rleigh@codelibre.net> wrote:
>
> > - with a lot of disc space
> How many TB?
>
> > - and a lot of spare CPU cycles
> Is sbuild able to scale linearly to 16-32 CPUs in a single VM or does it
> need many smaller servers?
> And how long would this take with e.g. 96 2.1 GHz Opteron CPU cores?

You can run several sbuild concurrently on the same node. In my setup, I
use sbuild with tarballs on tmpfs, and the main limiting factor for
scaling the number of concurrent sbuilds is RAM (which is still better
than I/O for me).

With enough machines/cores, the time for the whole rebuild is quickly
bound by the duration of the build for some big packages. In the past,
openoffice.org was the main blocker, but there are other "annoying"
packages now, such as openjdk-6 and insighttoolkit. When I don't build
the 4 packages that take the more time, a full archive rebuild
(including arch:all) takes ~4h.


Roger, it seems that you already got several offers, but if needed, I
could also help with that (I've done archive rebuilds for some DDs on a
quite regular basis).

- Lucas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110215111523.GA15479@xanadu.blop.info">http://lists.debian.org/20110215111523.GA15479@xanadu.blop.info


All times are GMT. The time now is 09:48 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.