FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 04-14-2010, 02:12 AM
Zac Medico
 
Default RESTRICT=parallel for builds that can't be executed in parallel

Hi everyone,

Should we add a RESTRICT=parallel value for ebuilds that can't be
built at the same time as other ebuilds? Brian says we need it for
things like xorg-server which calls eselect opengl.

If we truly need this, is RESTRICT=parallel a good name? We could
make it a PROPERTIES value instead, but then we'd probably end up
with something like PROPERTIES=no-parallel, so maybe RESTRICT is better.
--
Thanks,
Zac
 
Old 04-14-2010, 05:45 AM
Michał Górny
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On Tue, 13 Apr 2010 19:12:08 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> Should we add a RESTRICT=parallel value for ebuilds that can't be
> built at the same time as other ebuilds? Brian says we need it for
> things like xorg-server which calls eselect opengl.

I don't think that's the right solution. In most cases, xorg-server can
be built in parallel with stuff which doesn't require/set specific
opengl subsystem set.

Well, in fact is there _really_ any package which won't work with
switched opengl? I guess it's more of a runtime problem that buildtime,
and I don't think we really should print out loudly 'libGL has been
switched, please do not start OpenGL apps right now'.

In fact, the best solution in this particular case would be to patch
the buildsystem to use Gentoo location for particular OpenGL headers.

Disabling parallel emerge would be more of a workaround for the issue,
and will influence much more packages than it needs to. And it won't
help if user is running multiple emerge calls at the same time.

Another possible workaround is to enable some kind of 'eselect opengl'
locking so that another package requiring access to it will wait until
our build finishes. But this, of course, would require a quite good
solution for maintaining the lock and dropping it whenever build
process is aborted/killed.

--
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>
 
Old 04-14-2010, 06:10 AM
Brian Harring
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On Wed, Apr 14, 2010 at 07:45:20AM +0200, Michaaa GGGrny wrote:
> On Tue, 13 Apr 2010 19:12:08 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
>
> > Should we add a RESTRICT=parallel value for ebuilds that can't be
> > built at the same time as other ebuilds? Brian says we need it for
> > things like xorg-server which calls eselect opengl.
>
> I don't think that's the right solution. In most cases, xorg-server can
> be built in parallel with stuff which doesn't require/set specific
> opengl subsystem set.

RESTRICT=parallel is basically a big lock that forces building to go
down to one specific build/merge job- it's not at all fine grained.
That said, I'm not convinced it's worth actually *trying* to be fine
grained.

Stuff that needs this 'lock', if you look at it from the purely
academic angle is broken. The pkgs in question should be buildable
without modifying the livefs.

From the pragmatic angle, fixing some of those packages is a pretty
huge endeavour hence this lock existing. I see no reason to encourage
the usage of this lock by making it more fine grained, either.



> Well, in fact is there _really_ any package which won't work with
> switched opengl? I guess it's more of a runtime problem that buildtime,
> and I don't think we really should print out loudly 'libGL has been
> switched, please do not start OpenGL apps right now'.

Runtime and buildtime actually- consider a pkg that is mesa sensitive
having those headers/libs switched out mid build. That build likely
is going to boom in a rather interesting way.

Runtime itself, swapping out the gl resource that is used (going from
ati to x11 for building xorg-server) isn't going to make new apps that
start happy either.


> In fact, the best solution in this particular case would be to patch
> the buildsystem to use Gentoo location for particular OpenGL headers.

Academically, you're right, it's the proper solution. That takes a
fair amount of time however. More importantly, this issue *will* pop
up again elsewhere, meaning we'll have to delay the pkg in question
from being in the tree till it meets this higher QA standard.

Or we add this functionality, level the restrict, then go and fix the
ebuild- pragmatic solution.


> Disabling parallel emerge would be more of a workaround for the issue,
> and will influence much more packages than it needs to. And it won't
> help if user is running multiple emerge calls at the same time.

Running multiple emerges in parallel is already a bad idea. The
solution for that case is for the new/second emerge to feed the
request into the original emerge (or a daemon).

Keep in mind if support for multiple emerge invocations was
implemented it would still need some RESTRICT=parallel functionality
for screwed up pkgs like xorg-server. Fixing multiple emerge
invocations still requires fixing RESTRICT=parallel.


> Another possible workaround is to enable some kind of 'eselect opengl'
> locking so that another package requiring access to it will wait until
> our build finishes. But this, of course, would require a quite good
> solution for maintaining the lock and dropping it whenever build
> process is aborted/killed.

The other thing to recall is that by the time eselect is called, the
ebuilds environment may already be localized to settings that eselect
controls (LDPATH, that 'currently selected implementation' GL var).
My first thought is any locking scheme of that sort is going to be a
bigger can of worms.

~brian
 
Old 04-14-2010, 06:46 AM
Justin
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On 14/04/10 04:12, Zac Medico wrote:
> Hi everyone,
>
> Should we add a RESTRICT=parallel value for ebuilds that can't be
> built at the same time as other ebuilds? Brian says we need it for
> things like xorg-server which calls eselect opengl.

There is at least one other example which benefits from singlular build,
atlas libs. They run a benchmark suite to create platform specific
headers, which is heavily influenced by the system load. So having
RESTRICT=parallel would make the emerge more reliable.

justin
 
Old 04-14-2010, 07:06 AM
Duncan
 
Default RESTRICT=parallel for builds that can't be executed in parallel

Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted:

> RESTRICT=parallel is basically a big lock that forces building to go
> down to one specific build/merge job- it's not at all fine grained. That
> said, I'm not convinced it's worth actually *trying* to be fine grained.
>
> Stuff that needs this 'lock', if you look at it from the purely academic
> angle is broken. The pkgs in question should be buildable without
> modifying the livefs.
>
> From the pragmatic angle, fixing some of those packages is a pretty huge
> endeavour hence this lock existing. I see no reason to encourage the
> usage of this lock by making it more fine grained, either.

What examples of the problem do we have, other than xorg-server due to
eselect opengl?

For just xorg-server, killing parallel seems like a rather frustrating and
indeed broken solution (especially for folks who prefer to run freedomware
and thus have only the X11/mesa opengl version on their system anyway, so
forcing non-parallel is an exercise in uselessness). If it's the only
one, at /least/ only forcing non-parallel if the eselect opengl actually
changes the selected opengl would seem reasonable.

But if there's other non-contrived examples around, getting a couple of
them on the table should I think clarify our potential usage constraints
somewhat.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 04-14-2010, 09:02 AM
Brian Harring
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On Wed, Apr 14, 2010 at 07:06:56AM +0000, Duncan wrote:
> Brian Harring posted on Tue, 13 Apr 2010 23:10:16 -0700 as excerpted:
>
> > RESTRICT=parallel is basically a big lock that forces building to go
> > down to one specific build/merge job- it's not at all fine grained. That
> > said, I'm not convinced it's worth actually *trying* to be fine grained.
> >
> > Stuff that needs this 'lock', if you look at it from the purely academic
> > angle is broken. The pkgs in question should be buildable without
> > modifying the livefs.
> >
> > From the pragmatic angle, fixing some of those packages is a pretty huge
> > endeavour hence this lock existing. I see no reason to encourage the
> > usage of this lock by making it more fine grained, either.
>
> What examples of the problem do we have, other than xorg-server due to
> eselect opengl?

Honestly it's the only hard class of examples I've got- ati-drivers
(presumably nvidia also although I've not checked) also would require
this. That said, 7 years of pkging crap, I know it's not going to be
the *only* example.

It mostly comes down the perfect world view vs pragmatic- a lot of
what we're doing in terms of ebuild pkging is a balance between best
practice and having the thing usable.

Sometimes you've got to do something ugly to make it work- especially
when the effort required to do it correctly is fairly large. I'm not
arguing against doing it correctly, I'm just arguing we should be
realistic.


> For just xorg-server, killing parallel seems like a rather frustrating and
> indeed broken solution (especially for folks who prefer to run freedomware
> and thus have only the X11/mesa opengl version on their system anyway, so
> forcing non-parallel is an exercise in uselessness). If it's the only
> one, at /least/ only forcing non-parallel if the eselect opengl actually
> changes the selected opengl would seem reasonable.

The thing people need to keep in mind for stuff like this is that
metadata is *constant*- it's quite a huge amount of work to write a
framework that is able to ask "hey xorg, are you going to go screwing
w/ something that means we can't build something in parallel to you
building?"- it's not horrible, but it's a lot of effort for
questionable gain. If you consider users running ati-drivers, that
check would state "yes, lock it" for building both ati-drivers and
xorg-server; the only instance where it wouldn't trigger the lock is
when the user is running *just* xorg-server.

Also, just to be clear, RESTRICT=parallel does not means that building
xorg-server is going to be make -j1; it can still be make -jN, it just
keeps the scheduler from building any other jobs in parallel while
xorg-server is doing it's thing.


> But if there's other non-contrived examples around, getting a couple of
> them on the table should I think clarify our potential usage constraints
> somewhat.

As mentioned, I've got nothing else concrete- mostly because I've not
had the time to look (I do know they're going to be there however).
One thing to keep in mind also is that when situations of this sort
pop up we need the functionality *before* it occurs. Adding it after
the fact is a PITA.

What I'd like to see w/ RESTRICT=parallel is that it basically is
unused in the tree- I expect a few screwed up pkgs to need it however.
It's a bit like RESTRICT=sandbox; it should only be used when there is
an extremely good reason and the alternatives don't easily exist.
~harring
 
Old 04-14-2010, 12:58 PM
Michał Górny
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On Wed, 14 Apr 2010 08:46:15 +0200 Justin <jlec@gentoo.org> wrote:

> There is at least one other example which benefits from singlular
> build, atlas libs. They run a benchmark suite to create platform
> specific headers, which is heavily influenced by the system load. So
> having RESTRICT=parallel would make the emerge more reliable.

And that's another example of working around broken ideas. There is
an uncountable number of possibilities of many different factors
affecting the system load, and thus the results of the benchmark.
Removing one of them would indeed help in many cases but in many other
it wouldn't make any difference, additionally slowing down the whole
system update process.

Please notice that in order to implement that correctly, emerge would
have to wait until all previously running emerges are done, and avoid
running further ones while the build process is running. While
in the case of xorg-server, the 'lock' is needed throughout the whole
build process, in your case it is needed only for a short amount
of time when the benchmark is being performed --- and the 'real' part
of building would unnecessarily block emerge.

If this is ever to be implemented, it totally needs to be
user-overridable. Else, we'll end up someday like Windows, forcing user
to reboot the system and perform the merge on a dedicated runlevel.

--
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>
 
Old 04-14-2010, 01:03 PM
Duncan
 
Default RESTRICT=parallel for builds that can't be executed in parallel

Justin posted on Wed, 14 Apr 2010 08:46:15 +0200 as excerpted:

> On 14/04/10 04:12, Zac Medico wrote:
>> Hi everyone,
>>
>> Should we add a RESTRICT=parallel value for ebuilds that can't be built
>> at the same time as other ebuilds? Brian says we need it for things
>> like xorg-server which calls eselect opengl.
>
> There is at least one other example which benefits from singlular build,
> atlas libs. They run a benchmark suite to create platform specific
> headers, which is heavily influenced by the system load. So having
> RESTRICT=parallel would make the emerge more reliable.

sci-libs/blas-atlas (and perhaps lapack-atlas, etc, too), right?
"Automatically tuned..."

Wow! Yeah, that sounds like a reasonable example. Sort of like the
kernel does for md/RAID-5 and 6 at boot, I'd guess, choosing the fastest
algorithm on the platform, only they're doing it during system runtime
when who /knows/ what else is running! Having a second highly
parallelizable (MAKEOPTS version) build go from config to build and its
load go from say .8 to 8. in the middle of those benchmarks /could/ screw
things up "just a little!"

Thanks. That's just the sort of additional practical example I was asking
for to try to get my mind around this. Excellent example as, unlike the
various xorg/mesa/drivers thing, it's pretty hard to argue "just code
around it", for this one. The only technical way out of it here would
seem to be to change the build-and-benchmark strategy itself, which would
rather defeat the "automatically tuned" bit entirely.

BTW, gcc seems to do some stage output comparing in its bootstrap
process. Is that all absolute code correctness, or is there some
performance benchmarking there that could benefit from this as well?

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 04-14-2010, 11:36 PM
Ryan Hill
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On Wed, 14 Apr 2010 13:03:10 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> BTW, gcc seems to do some stage output comparing in its bootstrap
> process. Is that all absolute code correctness, or is there some
> performance benchmarking there that could benefit from this as well?

It's all code correctness.


--
fonts, by design, by neglect
gcc-porting, for a fact or just for effect
wxwidgets @ gentoo EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662
 
Old 04-21-2010, 10:34 AM
Luca Barbato
 
Default RESTRICT=parallel for builds that can't be executed in parallel

On 04/14/2010 08:46 AM, Justin wrote:
> There is at least one other example which benefits from singlular build,
> atlas libs. They run a benchmark suite to create platform specific
> headers, which is heavily influenced by the system load. So having
> RESTRICT=parallel would make the emerge more reliable.

I wonder if that case shouldn't be handled better with an huge ewarn so
people concerned would really run it in a benchmark environment, alone.

lu

--

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero
 

Thread Tools




All times are GMT. The time now is 06:51 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org