FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 11-26-2011, 02:22 PM
Michael Mol
 
Default emerge -j, make -j and make -l

I just wanted to share an experience I had today with optimizing parallel builds after discovering "-l" for Make...


I've got a little more tweaking I still want to do, but this is pretty awesome...


http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/


ZZ
 
Old 11-26-2011, 02:34 PM
Alex Schuster
 
Default emerge -j, make -j and make -l

Michael Mol writes:

> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/

404, but http://funnybutnot.wordpress.com/2011/11/26/ seems to work.

Wonko
 
Old 11-26-2011, 02:35 PM
"Stefan G. Weichinger"
 
Default emerge -j, make -j and make -l

Am 26.11.2011 16:22, schrieb Michael Mol:
> I just wanted to share an experience I had today with optimizing
> parallel builds after discovering "-l" for Make...
>
> I've got a little more tweaking I still want to do, but this is pretty
> awesome...
>
> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/

URL ... not found
 
Old 11-26-2011, 02:51 PM
Michael Mol
 
Default emerge -j, make -j and make -l

On Sat, Nov 26, 2011 at 10:34 AM, Alex Schuster <wonko@wonkology.org> wrote:
> Michael Mol writes:
>
>> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
>
> 404, but http://funnybutnot.wordpress.com/2011/11/26/ seems to work.

Should work now. Somehow, I triggered WP's "schedule a post" behavior.
Might still come up as 404 if you've got caching which caches
negatives.


--
:wq
 
Old 11-26-2011, 03:03 PM
"Stefan G. Weichinger"
 
Default emerge -j, make -j and make -l

Am 26.11.2011 16:34, schrieb Alex Schuster:
> Michael Mol writes:
>
>> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
>
> 404, but http://funnybutnot.wordpress.com/2011/11/26/ seems to work.

Yes, got it now as well.

Thanks for "quoting" me, Michael ... but I also googled that command
somewhere ... not my idea ... ;-)

Stefan
 
Old 11-27-2011, 08:27 AM
Mick
 
Default emerge -j, make -j and make -l

On Saturday 26 Nov 2011 15:22:15 Michael Mol wrote:
> I just wanted to share an experience I had today with optimizing parallel
> builds after discovering "-l" for Make...
>
> I've got a little more tweaking I still want to do, but this is pretty
> awesome...
>
> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
>
> ZZ

Thanks for sharing! How do you determine the optimum value for -l?
--
Regards,
Mick
 
Old 11-27-2011, 09:09 AM
Michael Mol
 
Default emerge -j, make -j and make -l

On Sun, Nov 27, 2011 at 4:27 AM, Mick <michaelkintzios@gmail.com> wrote:
> On Saturday 26 Nov 2011 15:22:15 Michael Mol wrote:
>> I just wanted to share an experience I had today with optimizing parallel
>> builds after discovering "-l" for Make...
>>
>> I've got a little more tweaking I still want to do, but this is pretty
>> awesome...
>>
>> http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
>>
>> ZZ
>
> Thanks for sharing! *How do you determine the optimum value for -l?

I'm making an educated guess. >.>

I figure that the optimal number of simultaneous CPU-consuming
processes is going to be the number of CPU cores, plus enough to keep
the CPU occupied while others are blocked on I/O. That's the same
reasoning that drives the selection of a -j number, really.

If I read make's man page correctly, -l acts as a threshold, choosing
not to spawn an additional child process if the system load average is
above a certain value Since system load is a count of actively running
and ready-to-run processes, you want it to be very close to your
number of logical cores[1].

Since it's going to be a spot decision for Make as to whether or not
to spawn another child (if it hits its limit, it's not going to check
again until after one of its children returns), there will be many
race cases where the load average is high when it looks, but some
other processes will return shortly afterward.[2] That means adding a
process or two for a fudge factor.

That's a lot of guess, though, and it still comes down to guess-and-check.

emerge -j8 @world # MAKEOPTS="-j16 -l10"

Was the first combination I tried. This completed in 89 minutes.

emerge -j8 @world # MAKEOPT="-j16 -l8"

Was the second. This took significantly longer.

I haven't tried higher than -l10; I needed this box to do be able to
do things, which meant installing more software. I've gone from 177
packages to 466.

[1] I don't have a hyperthreading system available, but I suspect that
this is also going to be true of logical cores; It's my understanding
that the overhead from overcommitting CPU comes primarily from context
switching between processors, and hyperthreading adds CPU hardware
specifically to reduce the need to context-switch in splitting
physical CPU resources between threads/processes. So while you'd lose
a little speed for an individual thread, you would gain it back in
aggregate over both threads.

[2] There would also be cases where the load average is low, such as
if a Make recipe calls for a significant bit of I/O before it consumes
a great deal of CPU, but a simple 7200rpm SATA disk appears to be
sufficiently fast that this case is less frequent.
--
:wq
 
Old 11-27-2011, 09:39 AM
 
Default emerge -j, make -j and make -l

Mick <michaelkintzios@gmail.com> wrote:

> On Saturday 26 Nov 2011 15:22:15 Michael Mol wrote:
> > I just wanted to share an experience I had today with optimizing parallel
> > builds after discovering "-l" for Make...
> >
> > I've got a little more tweaking I still want to do, but this is pretty
> > awesome...
> >
> > http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
> >
> > ZZ
>
> Thanks for sharing! How do you determine the optimum value for -l?

How do you get emerge not to display number of jobs and load average --
I only want to compile one at a time -- much safer that way and it is
doing that, but now it displays all that load average and how many jobs,
etc. -- any way to get rid of that display?

--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?

John Covici
covici@ccs.covici.com
 
Old 11-27-2011, 09:52 AM
Michael Mol
 
Default emerge -j, make -j and make -l

On Sun, Nov 27, 2011 at 5:39 AM, <covici@ccs.covici.com> wrote:
> Mick <michaelkintzios@gmail.com> wrote:
>
>> On Saturday 26 Nov 2011 15:22:15 Michael Mol wrote:
>> > I just wanted to share an experience I had today with optimizing parallel
>> > builds after discovering "-l" for Make...
>> >
>> > I've got a little more tweaking I still want to do, but this is pretty
>> > awesome...
>> >
>> > http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/
>> >
>> > ZZ
>>
>> Thanks for sharing! *How do you determine the optimum value for -l?
>
> How do you get emerge not to display number of jobs and load average --

The display is something emerge will show you if you've asked it to
build in parallel (which you did, by passing -j to emerge. That's
different from putting -j in MAKEOPTS.)

> I only want to compile one at a time -- much safer that way and it is
> doing that,

It's likely only doing that because there isn't anything it can
immediately build that doesn't have what it's *currently* working on
as a build dependency. I noted in my blog post that emerge's
parallelization has many of the same limitations as make's that's one
of the things I was talking about; there can be linchpin and keystone
packages which need to be built before many others. libc would be an
example. gcc is a frequent example.

> but now it displays all that load average and how many jobs,
> etc. -- any way to get rid of that display?

Forget the display; it sounds like you don't want emerge building in
parallel. In that event, don't pass "-j" to emerge. The display will
go away.

--
:wq
 
Old 11-27-2011, 09:58 AM
Dale
 
Default emerge -j, make -j and make -l

covici@ccs.covici.com wrote:

Mick<michaelkintzios@gmail.com> wrote:


On Saturday 26 Nov 2011 15:22:15 Michael Mol wrote:

I just wanted to share an experience I had today with optimizing parallel
builds after discovering "-l" for Make...

I've got a little more tweaking I still want to do, but this is pretty
awesome...

http://funnybutnot.wordpress.com/2011/11/26/optimizing-parallel-builds/

ZZ

Thanks for sharing! How do you determine the optimum value for -l?

How do you get emerge not to display number of jobs and load average --
I only want to compile one at a time -- much safer that way and it is
doing that, but now it displays all that load average and how many jobs,
etc. -- any way to get rid of that display?



Thank Zac for that. He thinks he knows what you want. ;-) Apparently
not huh?


http://forums.gentoo.org/viewtopic-t-901858.html

Just add --quiet-build=n to EMERGE_DEFAULT_OPTS in make.conf and it will
do it the old way.


Hope that helps.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 

Thread Tools




All times are GMT. The time now is 09:49 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org