EJOBS variable for EAPI 5?
 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote:
> On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
>> On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller
>> <ulm@gentoo.org> wrote:
>>> Coming back to this old topic [1]. Is there still consensus
>>> that we should have such an EJOBS variable? (It shouldn't be
>>> called JOBS because this name is too generic, see the old
>>> discussion.) Then we could add it to EAPI 5.
>>>
>>> Ulrich
>>>
>>> [1]
>>> <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
>>
>>
>>>
If we're doing this, do we tell users to stop setting MAKEOPTS for
>> EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs
>> 5 and greater instead? Do we put fancy code in the package
>> mangler to deal with it?
>
> Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS
> will have no effect for <EAPI5 ebuilds, then obviously we should
> not be advising users to stop using MAKEOPTS until the whole tree
> has migrated to EAPI5. And if EJOBS will be recognized by a future
> version of portage for all EAPIs, then we still should allow
> MAKEOPTS because some users may want to use --load-average.
>
> Changing the name of MAKEOPTS in >=EAPI5 makes no sense. First,
> because it's a standard environment variable used by gnu make.
> Second, because having 3 different settings for parallel building
> (EJOBS, MAKEOPTS, and "MAKEOPTS_EAPI5") would be insane.
>
> Fancy code in the package manager would be the way to go IMHO.
> Ulrich's message contains a reasonable description of the
> algorithm.
>
> -Alexandre.

I think, if i read the previous response to this correctly, that the
suggestion isn't the removal of MAKEOPTS, but simply that the '-j'
specification currently set in MAKEOPTS should instead be set in EJOBS
in everyone's make.conf. This would then be appended to MAKEOPTS (for
all EAPI) -and- be used for non-make build systems (for EAPI>=5) alike.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBA4YUACgkQ2ugaI38ACPD96gD+Pu9f9SVG//0yhioO0LGP/W8o
sIGpiMFIEddXvhUsDAwA/0EJkZF8jrN7zmt/LdZy3nlCGKTIkPNxp5ukUGDDWIJB
=Dlem
-----END PGP SIGNATURE-----

EJOBS variable for EAPI 5?
 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/08/12 12:08 PM, Ian Stakenvicius wrote:
> On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote:
>> On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
>>> On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller
>>> <ulm@gentoo.org> wrote:
>>>> Coming back to this old topic [1]. Is there still consensus
>>>> that we should have such an EJOBS variable? (It shouldn't be
>>>> called JOBS because this name is too generic, see the old
>>>> discussion.) Then we could add it to EAPI 5.
>>>>
>>>> Ulrich
>>>>
>>>> [1]
>>>> <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
>>>
>>>
>>>>
>
>>>>
If we're doing this, do we tell users to stop setting MAKEOPTS for
>>> EAPIs 5 and greater? Do we change the name of MAKEOPTS for
>>> EAPIs 5 and greater instead? Do we put fancy code in the
>>> package mangler to deal with it?
>
>> Users typically set MAKEOPTS systemwide in /etc/make.conf. If
>> EJOBS will have no effect for <EAPI5 ebuilds, then obviously we
>> should not be advising users to stop using MAKEOPTS until the
>> whole tree has migrated to EAPI5. And if EJOBS will be recognized
>> by a future version of portage for all EAPIs, then we still
>> should allow MAKEOPTS because some users may want to use
>> --load-average.
>
>> Changing the name of MAKEOPTS in >=EAPI5 makes no sense. First,
>> because it's a standard environment variable used by gnu make.
>> Second, because having 3 different settings for parallel
>> building (EJOBS, MAKEOPTS, and "MAKEOPTS_EAPI5") would be
>> insane.
>
>> Fancy code in the package manager would be the way to go IMHO.
>> Ulrich's message contains a reasonable description of the
>> algorithm.
>
>> -Alexandre.
>
> I think, if i read the previous response to this correctly, that
> the suggestion isn't the removal of MAKEOPTS, but simply that the
> '-j' specification currently set in MAKEOPTS should instead be set
> in EJOBS in everyone's make.conf. This would then be appended to
> MAKEOPTS (for all EAPI) -and- be used for non-make build systems
> (for EAPI>=5) alike.
>


..hit send to soon...

So if users stick with setting -j in MAKEOPTS, then in EAPI=5 and
above this would only affect make-based builds; for parallel
compilation in non-make builds they would need to convert to using
EJOBS for EAPI=5 and above, otherwise those build systems will compile
serially instead of in parallel.





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBA4jQACgkQ2ugaI38ACPAFrAEArp7MM5w4Mv/TLKb058HzB9oN
NtQeSVoCQ8X5PuxjjJ0BAKbTJXEkLlZ0hMr09RyTKzK0XtdQq6 cf2fbeFFgFb5eV
=+vtW
-----END PGP SIGNATURE-----

EJOBS variable for EAPI 5?
 

On 09/04/2012 04:00 AM, Walter Dnes wrote:
> On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote
>
>> This approach is fine imo, although I'd *potentially* look at adding a
>> magic $PROC_COUNT var that is the # of cpu threads on the system;
>> either that or defaulting jobs to it.
>>
>> I rather dislike requiring users to go jam a 2/4/8 in there when it's
>> easy to compute. That said, it's minor.
>>
>> Either way, yes, I think EJOBS should be in EAPI5.
>
> One question about the suggested EJOBS variable; will it over-ride
> MAKEOPTS? Every so often on the Gentoo-user list, someone comes along
> with a mysterious build failure. The first suggestion is to reset
> MAKEOPTS to -j1. And on some occasions, that is indeed the solution to
> the mysterious build failure.

That would be due to a missing dependency in the Makefiles, and using
-j1 is just a workaround. The ebuild can be hardcoded to use emake -j1
until the Makefile gets fixed.

> I set -j1 and leave it that way. Yes, the builds take longer, but the
> resulting binary is just as fast. And the amount of time I "save" will
> be blown away the first time I end up screwing around a couple of hours
> trying to fix a mysterious build failure. That's why I want the user to
> have the option of over-riding EJOBS, should it ever be implemented.

You could use EXTRA_EMAKE for that. You can do per-package settings via
/etc/portage/package.env.
--
Thanks,
Zac

EJOBS variable for EAPI 5?
 

Il 04/09/2012 19:15, Zac Medico ha scritto:

On 09/04/2012 04:00 AM, Walter Dnes wrote:

On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote


This approach is fine imo, although I'd *potentially* look at adding a
magic $PROC_COUNT var that is the # of cpu threads on the system;
either that or defaulting jobs to it.

I rather dislike requiring users to go jam a 2/4/8 in there when it's
easy to compute. That said, it's minor.

Either way, yes, I think EJOBS should be in EAPI5.

One question about the suggested EJOBS variable; will it over-ride
MAKEOPTS? Every so often on the Gentoo-user list, someone comes along
with a mysterious build failure. The first suggestion is to reset
MAKEOPTS to -j1. And on some occasions, that is indeed the solution to
the mysterious build failure.

That would be due to a missing dependency in the Makefiles, and using
-j1 is just a workaround. The ebuild can be hardcoded to use emake -j1
until the Makefile gets fixed.


I set -j1 and leave it that way. Yes, the builds take longer, but the
resulting binary is just as fast. And the amount of time I "save" will
be blown away the first time I end up screwing around a couple of hours
trying to fix a mysterious build failure. That's why I want the user to
have the option of over-riding EJOBS, should it ever be implemented.

You could use EXTRA_EMAKE for that. You can do per-package settings via
/etc/portage/package.env.


Dunno where to place this request, but if we go for something like EJOBS
could we also make it phase specific?

So compile, install and test could have a different number of jobs running.
Possibly three different variables that override a predefined EJOBS.

TIA

EJOBS variable for EAPI 5?
 

On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
> Dunno where to place this request, but if we go for something like EJOBS
> could we also make it phase specific?
> So compile, install and test could have a different number of jobs running.
> Possibly three different variables that override a predefined EJOBS.

Per-phase sounds a little to fine-grained. Instead, I'd suggest to add
an ELOADAVG variable that's analogous to make's --load-average option.
That should be enough to compensate for any differences between phases.
--
Thanks,
Zac

EJOBS variable for EAPI 5?
 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/09/12 12:43 PM, Zac Medico wrote:
> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>> Dunno where to place this request, but if we go for something
>> like EJOBS could we also make it phase specific? So compile,
>> install and test could have a different number of jobs running.
>> Possibly three different variables that override a predefined
>> EJOBS.
>
> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
> add an ELOADAVG variable that's analogous to make's --load-average
> option. That should be enough to compensate for any differences
> between phases.

I personally wonder about why this would be necessary from the
perspective of the user; if the user's system at emerge time can
handle X concurrent processes per emerge-job , i don't see why it
would matter what phase these jobs would be launched from.

At the ebuild level, certainly, but that's one of the reasons for
EJOBS in the first place, so that it can be overridden consistently
within a phase, if necessary for the ebuild (regardless of build
system type), right?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBPbLEACgkQ2ugaI38ACPA1qAD/bvjy7aB6nk5YboJHnCpQ8C56
QolKD9BPHL9eN8Xf41oA/iTZU+tyB+BDl+woZAlVGbaa6AR2r6Qp8rwOzkUWSwV/
=FhKc
-----END PGP SIGNATURE-----

EJOBS variable for EAPI 5?
 

On 09/11/2012 09:54 AM, Ian Stakenvicius wrote:
> On 11/09/12 12:43 PM, Zac Medico wrote:
>> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>>> Dunno where to place this request, but if we go for something
>>> like EJOBS could we also make it phase specific? So compile,
>>> install and test could have a different number of jobs running.
>>> Possibly three different variables that override a predefined
>>> EJOBS.
>
>> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
>> add an ELOADAVG variable that's analogous to make's --load-average
>> option. That should be enough to compensate for any differences
>> between phases.
>
> I personally wonder about why this would be necessary from the
> perspective of the user; if the user's system at emerge time can
> handle X concurrent processes per emerge-job , i don't see why it
> would matter what phase these jobs would be launched from.

Right, what matters is the system load, which is why I suggested ELOADAVG.

> At the ebuild level, certainly, but that's one of the reasons for
> EJOBS in the first place, so that it can be overridden consistently
> within a phase, if necessary for the ebuild (regardless of build
> system type), right?

Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS
though, since overloading is not a good idea, and can happen easily any
time that you doing lots of things in parallel.
--
Thanks,
Zac

EJOBS variable for EAPI 5?
 

On Tue, Sep 11, 2012 at 1:07 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 09/11/2012 09:54 AM, Ian Stakenvicius wrote:
>> At the ebuild level, certainly, but that's one of the reasons for
>> EJOBS in the first place, so that it can be overridden consistently
>> within a phase, if necessary for the ebuild (regardless of build
>> system type), right?
>
> Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS
> though, since overloading is not a good idea, and can happen easily any
> time that you doing lots of things in parallel.

I tend to agree that load average matters more, although that doesn't
factor in RAM use.

I don't suggest that this is something that is easily remedied, but I
have run into a situation where WHAT you're doing greatly influences
how many jobs you can run - distcc. I once tried to implement distcc
in a fairly large cluster, and then run make with VERY high levels of
parallelization - such as -j32. That worked great - if the package
used C. Then portage would try to build something that used ant and
suddenly the host was trying to run 32 jvms in parallel - just killing
the system. Ditto for python/etc. There was no way to tell the build
system to go nuts with anything using distcc but not everything else.

I think this is just a fundamental limitation of make - I don't
suggest that we can fix this at the portage level. However, that is a
use case where WHAT you're doing influences how many jobs you can run.

Rich

EJOBS variable for EAPI 5?
 

On 9/11/2012 9:54 AM, Ian Stakenvicius wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/09/12 12:43 PM, Zac Medico wrote:

On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:

Dunno where to place this request, but if we go for something
like EJOBS could we also make it phase specific? So compile,
install and test could have a different number of jobs running.
Possibly three different variables that override a predefined
EJOBS.


Per-phase sounds a little to fine-grained. Instead, I'd suggest to
add an ELOADAVG variable that's analogous to make's --load-average
option. That should be enough to compensate for any differences
between phases.


I personally wonder about why this would be necessary from the
perspective of the user; if the user's system at emerge time can
handle X concurrent processes per emerge-job , i don't see why it
would matter what phase these jobs would be launched from.

At the ebuild level, certainly, but that's one of the reasons for
EJOBS in the first place, so that it can be overridden consistently
within a phase, if necessary for the ebuild (regardless of build
system type), right?


I would hope so. There are definitely real-world reasons to want to
restrict jobs -- usually to just one -- in a particular phase... several
ebuilds, notably several in @system, do this already by injecting -j1
somewhere.


To say it's all about performance seems to be forgetting a major reason
somebody might want to do this as a user or an ebuild author.


Hmm...

I should preface this by saying that it really pains me to say this,
because now this just starts to seem so fucking complicated -- but now
that I think about it, this really seems to highlight a kind of semantic
discrepancy that this thread has mostly dusted under the rug.


There are really TWO things we seem to care about -- one is some kind of
global build-parallelism frob-set, and the other is the ability to turn
off all parallelism for certain parts of an ebuild and those are
actually kind-of orthogonal.


In other words, imagine if we had some kind of "global parallellism
arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other
things we haven't thought of yet, and decided things like:


o should portage start a new parallel emerge process?
o What is the appropriate "MAKEOPTS" for the "emake"
that foo-1.0.ebuild just issued?

or even, one could imagine, such things as:

o should portage decide that parallelism has gotten out of control
somehow and suspend or kill a running ebuild for subsequent
resuming or restarting?

o should portage re-nice some of its processes?

Note that, effectively, we have this already, and it's called "portage".
But one could certainly make a case for modularizing it better, since,
in truth, we are talking about a very common, very abstract problem here
which portage shares with any number of batch-build systems.


Such an engine could very well do exactly the right thing if it were
faced with a constraint that a certain part of a certain build needed to
proceed without parallelism due to limitations coming from the build.


Also, there are very large parts of most builds -- configure comes to
mind -- that don't parallelize even if, perhaps, they should. In such
cases, a really smart global parallelism arbiter could easily respond by
spawning more jobs from other builds.


Not sure what I'm suggesting we do about it, exactly, but just pointing
out that maybe a completely "correct" solution requires a much more
elaborate implementation than just a bunch of syntactic sugar around
what we currently call MAKEOPTS. Whether or not Gentoo wants to take
that all on, right now, as part of the next EAPI, is certainly debatable
-- in fact, I'm inclined to say maybe it's not the best idea.


Perhaps we should really be asking: is the status-quo really so
problematic/inelegant that it needs fixing? -- before we decide how best
to fix it.


-gmt

EJOBS variable for EAPI 5?
 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 12/09/12 05:55 AM, Gregory M. Turner wrote:
>
> Note that, effectively, we have this already, and it's called
> "portage". But one could certainly make a case for modularizing it
> better, since, in truth, we are talking about a very common, very
> abstract problem here which portage shares with any number of
> batch-build systems.
>
> Such an engine could very well do exactly the right thing if it
> were faced with a constraint that a certain part of a certain build
> needed to proceed without parallelism due to limitations coming
> from the build.
>
> Also, there are very large parts of most builds -- configure comes
> to mind -- that don't parallelize even if, perhaps, they should.
> In such cases, a really smart global parallelism arbiter could
> easily respond by spawning more jobs from other builds.
>

So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?

Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc). However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels. We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??).. It would be an interesting project, though.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBQhwAACgkQ2ugaI38ACPCKrgD+JNlHPUl7ET YDC6u3lYWRSz8J
fpWC/puDfCYu51yNOVIA/0E+U6x9Ds8GV8r/RinkTqss3/fcd06w24GRvZOda3Mj
=XONZ
-----END PGP SIGNATURE-----