FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 12-30-2007, 12:46 PM
Brendan O'Dea
 
Default Parallel build results

On Sun, Dec 30, 2007 at 02:17:05PM +0100, Michael Tautschnig wrote:
>I don't actually remember the part of this discussion where the real cause of
>the problem with the above rule was discussed, but in case it is about some
>dependencies between the rules (which is the most likely IMHO), then it should
>really be fixed as follows
>
>doc_install:
> ...
>
>pure_instal: doc_install
> ...
>
>all: pure_install
> ...
>
>install:: all
> ...

Thanks, not exactly what I need but sufficient to provide a solution:
push the "all" dependency from "install" down to each of the targets
which may be run in parallel.

install: pure_install doc_install
...

pure_install: all
...

doc_install: all
...

--bod


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-30-2007, 12:53 PM
Adam Borowski
 
Default Parallel build results

On Sun, Dec 30, 2007 at 11:43:00PM +1100, Brendan O'Dea wrote:
> On Sun, Dec 02, 2007 at 11:13:53PM +0000, Ben Hutchings wrote:
> >It appears that ExtUtils::MakeMaker, a standard Perl module commonly
> >used to generate Makefiles for Perl modules, emits the rule:
> >
> >install :: all pure_install doc_install

> install :: all
> $(MAKE) pure_install doc_install

Both are buggy, because neither makes sure any of the _install targets have
their files built. This means, trying to run "X_install" before or instead
of full "install" can break; that can happen if someone works on the package
or if there are separate binary-arch and binary-indep targets.

What about this:

install :: pure_install doc_install
pure_install: pure_build
doc_install: doc_build

(either pure_build or doc_build can be "all" if there's no easy way to
separate the main and doc builds)

--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-30-2007, 05:06 PM
Russ Allbery
 
Default Parallel build results

Brendan O'Dea <bod@debian.org> writes:
> On Sun, Dec 02, 2007 at 11:13:53PM +0000, Ben Hutchings wrote:

>> It appears that ExtUtils::MakeMaker, a standard Perl module commonly
>> used to generate Makefiles for Perl modules, emits the rule:
>>
>> install :: all pure_install doc_install
>>
>> This appears to account for the failure of some of my Perl packages,
>> and probably many others. This should be fixed in MakeMaker, not in
>> the packages that use it.
>
> About the only option I can see is to use something like:
>
> install :: all
> $(MAKE) pure_install doc_install
>
> which I'm not terribly keen on. I'm open to better suggestions.

As long as you also have:

pure_install:: all
doc_install:: all

won't make figure it out even with the first form? It may try to run all
three at the same time, but it will then realize that the second two
depend on all anyway, so all should be forced to go first and complete
before either of the other targets run.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-31-2007, 08:07 PM
Robert Millan
 
Default Parallel build results

On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
> I finally got through the test builds of all the source packages in sid for
> i386 using dpkg-buildpackage -j3 on a dual core machine. The results as
> before are at http://people.debian.org/~schepler/build-logs/bymaint.html .

Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try to
fix those first?

--
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call, if you are unable to speak?
(as seen on /.)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-31-2007, 11:13 PM
Adam Borowski
 
Default Parallel build results

On Mon, Dec 31, 2007 at 10:07:15PM +0100, Robert Millan wrote:
> On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
> > I finally got through the test builds of all the source packages in sid for
> > i386 using dpkg-buildpackage -j3 on a dual core machine. The results as
> > before are at http://people.debian.org/~schepler/build-logs/bymaint.html .
>
> Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try to
> fix those first?

Once you go over -j1, nearly all usual race conditions can be potentially
triggered. You would need a quite contrived build system to get a three-way
race which cannot fail with -j2.

And races caused by doing more than one task at once are pretty rare, in
most in the cases the failure is due to wrong dependences. The root of the
problem is, people are used to a single-tasked make which always builds
everything from left to right. Things won't break until -j or -k is used.

What about changing make so it builds prerequisites in a random order, or at
least right-to-left which is typically the worst case? This would help spot
dependency problems other than true races.


(an example which will not be caught by hypothethical make --order=random
but will likely break for -j higher than 1:

all: foo bar

foo: foo.c
gcc src.c
mv a.out $(target)/foo

bar: bar.c
gcc bar.c
mv a.out $(target)/bar

Another example could be building two flavours of the same package.)


True race conditions are non-deterministic, so even repeated test builds
are likely to find most of them, but wrong dependencies are a lot easier
to catch.

--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-01-2008, 11:29 PM
Aurelien Jarno
 
Default Parallel build results

On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
> I finally got through the test builds of all the source packages in sid for
> i386 using dpkg-buildpackage -j3 on a dual core machine. The results as
> before are at http://people.debian.org/~schepler/build-logs/bymaint.html .
> Some statistics:
>

> 1014 succeeded, but with jobserver warnings
>

No that DEB_BUILD_OPTIONS="parallel=n" has been superseded by
dpkg-buildpackage -j, I am trying to fix the glibc package so that it
can be built in parallel.

Unfortunately I failed to get it built on parallel (whereas with
DEB_BUILD_OPTIONS it was possible), I am getting:

make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.

This basically means only the debian/rules file is run in parallel, the
real build is not, whereas with DEB_BUILD_OPTIONS it was the contrary.
You can imagine the difference in term of build time.

This is due to the following command:
$(call logme, -a $(log_build), $(MAKE) -C $(DEB_BUILDDIR))

where logme is defined as:

define logme
(exec 3>&1; exit `( ( ( $(2) ) 2>&1 3>&-; echo $$? >&4) | tee $(1) >&3)
4>&1`)
endef

Any idea how to get this working for parallel builds with
dpkg-buildpackage -j? I fail to see a solution.

--
.'`. Aurelien Jarno | GPG: 1024D/F1BCDB73
: :' : Debian developer | Electrical Engineer
`. `' aurel32@debian.org | aurelien@aurel32.net
`- people.debian.org/~aurel32 | www.aurel32.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-01-2008, 11:36 PM
Aurelien Jarno
 
Default Parallel build results

On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
> I finally got through the test builds of all the source packages in sid for
> i386 using dpkg-buildpackage -j3 on a dual core machine. The results as
> before are at http://people.debian.org/~schepler/build-logs/bymaint.html .
> Some statistics:
>
> 204 built BROKEN packages
> 1408 FAILED
> 230 FAILED, even with regular build
> 8986 succeeded
> 1014 succeeded, but with jobserver warnings
>
> These are not encouraging statistics, especially considering the fact that
> there are undoubtedly many false negatives, so I'll hold off on submitting
> bug reports for now.

Did you compare the contents of the package with and without -j? I am
almost sure some of the successfully built packages are actually not
correctly built and some files are missing.

For example I remember having seen some python packages building one
flavour after the other in different targets. I imagine this could
result in one of the flavour being overwritten by the other, and thus
not present in the resulting .deb file.

--
.'`. Aurelien Jarno | GPG: 1024D/F1BCDB73
: :' : Debian developer | Electrical Engineer
`. `' aurel32@debian.org | aurelien@aurel32.net
`- people.debian.org/~aurel32 | www.aurel32.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-01-2008, 11:49 PM
Russ Allbery
 
Default Parallel build results

Aurelien Jarno <aurelien@aurel32.net> writes:
> On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:

>> I finally got through the test builds of all the source packages in sid
>> for i386 using dpkg-buildpackage -j3 on a dual core machine. The
>> results as before are at
>> http://people.debian.org/~schepler/build-logs/bymaint.html . Some
>> statistics:
>
>> 1014 succeeded, but with jobserver warnings
>
> No that DEB_BUILD_OPTIONS="parallel=n" has been superseded by
> dpkg-buildpackage -j, I am trying to fix the glibc package so that it
> can be built in parallel.

The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The
package should continue to look at DEB_BUILD_OPTIONS to determine whether
parallel builds should be done.

See http://bugs.debian.org/209008 which is awaiting final reviews and
seconds and should be in the next version of Policy, and the
dpkg-buildpackage man page under the -j option.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 12:02 AM
Aurelien Jarno
 
Default Parallel build results

Russ Allbery a écrit :
> Aurelien Jarno <aurelien@aurel32.net> writes:
>> On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
>
>>> I finally got through the test builds of all the source packages in sid
>>> for i386 using dpkg-buildpackage -j3 on a dual core machine. The
>>> results as before are at
>>> http://people.debian.org/~schepler/build-logs/bymaint.html . Some
>>> statistics:
>>> 1014 succeeded, but with jobserver warnings
>> No that DEB_BUILD_OPTIONS="parallel=n" has been superseded by
>> dpkg-buildpackage -j, I am trying to fix the glibc package so that it
>> can be built in parallel.
>
> The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The
> package should continue to look at DEB_BUILD_OPTIONS to determine whether
> parallel builds should be done.

As far as I understand, the main effect is to call debian/rules with -j,
but it also set DEB_BUILD_OPTIONS (but I fail to see why).

Anyway, if the package continue to look to DEB_BUILD_OPTIONS, as the
make children could not communicate with the jobserver, some strange
results may appear. On architectures that have more than one flavour,
that just mean the number of jobs is actually (value of -j times number
of flavours).

--
.'`. Aurelien Jarno | GPG: 1024D/F1BCDB73
: :' : Debian developer | Electrical Engineer
`. `' aurel32@debian.org | aurelien@aurel32.net
`- people.debian.org/~aurel32 | www.aurel32.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 12:07 AM
Russ Allbery
 
Default Parallel build results

Aurelien Jarno <aurelien@aurel32.net> writes:
> Russ Allbery a écrit :

>> The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The
>> package should continue to look at DEB_BUILD_OPTIONS to determine
>> whether parallel builds should be done.

> As far as I understand, the main effect is to call debian/rules with -j,
> but it also set DEB_BUILD_OPTIONS (but I fail to see why).

It also sets MAKEFLAGS. I'm not sure that part is a good idea, since
packages that don't anticipate this can't reject the provided MAKEFLAGS in
the case that the upstream build system doesn't support parallel builds.
Basically, this means that using dpkg-buildpackage -j isn't generally
safe. It can only be used with packages that you already know support
parallel builds.

> Anyway, if the package continue to look to DEB_BUILD_OPTIONS, as the
> make children could not communicate with the jobserver, some strange
> results may appear. On architectures that have more than one flavour,
> that just mean the number of jobs is actually (value of -j times number
> of flavours).

Ah, interesting. This is a subtlety of how make implements parallelism
that I wasn't aware of.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>
 

Thread Tools




All times are GMT. The time now is 12:10 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org