FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 01-02-2008, 01:07 AM
Aurelien Jarno
 
Default Parallel build results

Russ Allbery a écrit :
> Aurelien Jarno <aurelien@aurel32.net> writes:
>> Russ Allbery a écrit :
>
>>> The effect of dpkg-buildpackage -j is to set DEB_BUILD_OPTIONS. The
>>> package should continue to look at DEB_BUILD_OPTIONS to determine
>>> whether parallel builds should be done.
>
>> As far as I understand, the main effect is to call debian/rules with -j,
>> but it also set DEB_BUILD_OPTIONS (but I fail to see why).
>
> It also sets MAKEFLAGS. I'm not sure that part is a good idea, since
> packages that don't anticipate this can't reject the provided MAKEFLAGS in
> the case that the upstream build system doesn't support parallel builds.
> Basically, this means that using dpkg-buildpackage -j isn't generally
> safe. It can only be used with packages that you already know support
> parallel builds.
>

I think that is a problem. dpkg-buildpackage -j will be really usable
when *all* packages support parallel builds.

On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages
that have not been validated by the maintainers, and used by packages
that have been tested by the maintainer. Also it was possible to use
only on some parts of the build. For example the make install part is
sometimes problematic, and mainly depends on I/O throughput of the
machine, and not CPU. This make some problems difficult to detect, as
they happen only in rare case on some machines.

Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call
debian/rules with -j and the value it founds in this environment
variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the
same effect as dpkg-buildpackage -j.

I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some
packages is a first goal to achieve. Supporting this option in the 100
packages which take the longest to build would probably be a huge
improvement in build time of the whole archive, while being easier than
fixing all the packages in the archive. The most difficult part probably
being changing the policy (sigh) as some of them already support that,
but through a specific environment variable. Also a lot of packages
spend most of their time in the configure script for those using
autotools, and in installing/removing build-depends packages for the
others. Using make -j in those cases won't help.

--
.'`. Aurelien Jarno | GPG: 1024D/F1BCDB73
: :' : Debian developer | Electrical Engineer
`. `' aurel32@debian.org | aurelien@aurel32.net
`- people.debian.org/~aurel32 | www.aurel32.net


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 01:23 AM
Russ Allbery
 
Default Parallel build results

Aurelien Jarno <aurelien@aurel32.net> writes:

> On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages
> that have not been validated by the maintainers, and used by packages
> that have been tested by the maintainer. Also it was possible to use
> only on some parts of the build. For example the make install part is
> sometimes problematic, and mainly depends on I/O throughput of the
> machine, and not CPU. This make some problems difficult to detect, as
> they happen only in rare case on some machines.
>
> Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call
> debian/rules with -j and the value it founds in this environment
> variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the
> same effect as dpkg-buildpackage -j.
>
> I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some
> packages is a first goal to achieve. Supporting this option in the 100
> packages which take the longest to build would probably be a huge
> improvement in build time of the whole archive, while being easier than
> fixing all the packages in the archive.

I agree with this analysis. I think the MAKEFLAGS setting in
dpkg-buildpackage should be removed and instead calling dpkg-buildpackage
with the -j option should just set the parallel=n flag in
DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to
decide how to implement this.

If someone *really* wants to try forcing make to do a parallel build, they
can always set MAKEFLAGS themselves directly.

The only drawback that I can see is that, without special intervention by
the package, this doesn't run debian/rules itself in parallel. But I
think the latter generates a lot of bugs and doesn't save a lot of time in
practice, so I'm okay with that.

> The most difficult part probably being changing the policy (sigh) as
> some of them already support that, but through a specific environment
> variable.

My intention is to have parallel=n be specified in Policy 3.7.4. If you
have a moment to review the wording in Bug#209008, feedback and seconds
are certainly welcome.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 12:23 PM
Lucas Nussbaum
 
Default Parallel build results

On 01/01/08 at 18:23 -0800, Russ Allbery wrote:
> Aurelien Jarno <aurelien@aurel32.net> writes:
>
> > On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages
> > that have not been validated by the maintainers, and used by packages
> > that have been tested by the maintainer. Also it was possible to use
> > only on some parts of the build. For example the make install part is
> > sometimes problematic, and mainly depends on I/O throughput of the
> > machine, and not CPU. This make some problems difficult to detect, as
> > they happen only in rare case on some machines.
> >
> > Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call
> > debian/rules with -j and the value it founds in this environment
> > variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the
> > same effect as dpkg-buildpackage -j.
> >
> > I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some
> > packages is a first goal to achieve. Supporting this option in the 100
> > packages which take the longest to build would probably be a huge
> > improvement in build time of the whole archive, while being easier than
> > fixing all the packages in the archive.
>
> I agree with this analysis. I think the MAKEFLAGS setting in
> dpkg-buildpackage should be removed and instead calling dpkg-buildpackage
> with the -j option should just set the parallel=n flag in
> DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to
> decide how to implement this.

I filed bug #458589 against dpkg-dev about this. I think that both
following solutions are fine:

- dpkg-buildpackage must not consider DEB_BUILD_OPTIONS=parallel=n as
the same thing as calling dpkg-buildpackage -j n.

- dpkg-buildpackage -j must not set MAKEFLAGS. But if this is choosen,
dpkg-buildpackage -j will simply become another way of doing 'export
DEB_BUILD_OPTIONS="parallel=n"'. Is it really worth keeping in this
case?

> If someone *really* wants to try forcing make to do a parallel build, they
> can always set MAKEFLAGS themselves directly.

True ; maybe dpkg-buildpackage -j is completely useless, actually.
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.net http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr GPG: 1024D/023B3F4F |


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 04:21 PM
Daniel Schepler
 
Default Parallel build results

On Tuesday 01 January 2008 07:36:34 pm Aurelien Jarno wrote:
> Did you compare the contents of the package with and without -j? I am
> almost sure some of the successfully built packages are actually not
> correctly built and some files are missing.
>
> For example I remember having seen some python packages building one
> flavour after the other in different targets. I imagine this could
> result in one of the flavour being overwritten by the other, and thus
> not present in the resulting .deb file.

I compared the control files and file lists from the packages and marked the
build "broken" if it found significant differences. So that would have
caught the case above. But not a case where, for example, the compiler runs
were started before patches were completely applied, probably resulting in a
mix of patched and non-patched code.

However, due to the issues raised in the previous thread about 100% bit-by-bit
reproducibility of package builds, I don't see any good way to detect that
case automatically.
--
Daniel Schepler


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-02-2008, 04:27 PM
Daniel Schepler
 
Default Parallel build results

On Monday 31 December 2007 04:07:15 pm Robert Millan wrote:
> On Sat, Dec 01, 2007 at 09:21:33PM -0500, Daniel Schepler wrote:
> > I finally got through the test builds of all the source packages in sid
> > for i386 using dpkg-buildpackage -j3 on a dual core machine. The results
> > as before are at
> > http://people.debian.org/~schepler/build-logs/bymaint.html .
>
> Why -j3 ? It's already quite an effort to aim for -j2. Shouldn't we try
> to fix those first?

That's just my habit, the idea being to increase the probability that both
CPUs have work to do while I/O is going on.

And besides, I don't see that much of a qualitative difference between -j2
and -j3, for reasons already pointed out by others in the thread.
--
Daniel Schepler


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 01-03-2008, 06:44 AM
Guillem Jover
 
Default Parallel build results

Hi,

On Wed, 2008-01-02 at 14:23:01 +0100, Lucas Nussbaum wrote:
> On 01/01/08 at 18:23 -0800, Russ Allbery wrote:
> > Aurelien Jarno <aurelien@aurel32.net> writes:
> > > On the other hand, DEB_BUILD_OPTIONS=parallel=n was ignored by packages
> > > that have not been validated by the maintainers, and used by packages
> > > that have been tested by the maintainer. Also it was possible to use
> > > only on some parts of the build. For example the make install part is
> > > sometimes problematic, and mainly depends on I/O throughput of the
> > > machine, and not CPU. This make some problems difficult to detect, as
> > > they happen only in rare case on some machines.

Well on my part I think I've made all my packages to properly build in
parallel but didn't add the explicit parsing on the rules file.

> > > Unfortunately dpkg-buildpackage now looks for DEB_BUILD_OPTIONS and call
> > > debian/rules with -j and the value it founds in this environment
> > > variable. That makes DEB_BUILD_OPTIONS=parallel=n useless, as it has the
> > > same effect as dpkg-buildpackage -j.

No, there's build systems out there that are not make based, those
would still need to parse DEB_BUILD_OPTIONS and activate parallelism
if supported by those systems.

> > > I personally think that supporting DEB_BUILD_OPTIONS=parallel=n on some
> > > packages is a first goal to achieve. Supporting this option in the 100
> > > packages which take the longest to build would probably be a huge
> > > improvement in build time of the whole archive, while being easier than
> > > fixing all the packages in the archive.
> >
> > I agree with this analysis. I think the MAKEFLAGS setting in
> > dpkg-buildpackage should be removed and instead calling dpkg-buildpackage
> > with the -j option should just set the parallel=n flag in
> > DEB_BUILD_OPTIONS. From there, we can leave it to package maintainers to
> > decide how to implement this.
>
> I filed bug #458589 against dpkg-dev about this. I think that both
> following solutions are fine:
>
> - dpkg-buildpackage must not consider DEB_BUILD_OPTIONS=parallel=n as
> the same thing as calling dpkg-buildpackage -j n.

Ok, I've just commited a change for dpkg-buildpackage not to automatically
behave as if -j had been passed as argument if DEB_BUILD_OPTIONS contains
parallel=n.

> - dpkg-buildpackage -j must not set MAKEFLAGS. But if this is choosen,
> dpkg-buildpackage -j will simply become another way of doing 'export
> DEB_BUILD_OPTIONS="parallel=n"'. Is it really worth keeping in this
> case?
>
> > If someone *really* wants to try forcing make to do a parallel build, they
> > can always set MAKEFLAGS themselves directly.
>
> True ; maybe dpkg-buildpackage -j is completely useless, actually.

It's as useless as dpkg-buildpackage itself, you can also directly
call debian/rules ...

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 04:16 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org