FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 04-14-2010, 02:20 PM
Michał Górny
 
Default RESTRICT=parallel for builds that can't be executed in parallel)

On Tue, 13 Apr 2010 23:10:16 -0700
Brian Harring <ferringb@gmail.com> wrote:

> Running multiple emerges in parallel is already a bad idea. The
> solution for that case is for the new/second emerge to feed the
> request into the original emerge (or a daemon).

Although such solution will be useful in many cases indeed, there are
still many advantages of having few separate emerge calls running
in parallel.

First of all, the 'emerge daemon' would have to be able to
transparently inject the additional packages onto the dependency tree.
And just appending missing dependencies and the real package at the end
doesn't seem sufficient.

For example, what if such operation would result in a conflict? If
that's only that simple as 'A and B can not be installed at the same
time', the second emerge call could reject appending the packages,
explaining the reason for that.

But what if the conflict could be resolved through pulling in some
other version of one of the packages? We could try to replace
the package in the dependency tree silently if it didn't yet start
being merged; but what if it did? Should we reject adding
the packages, encouraging user to 'try again later'? Or maybe abort
the running builds to merge another version of the depend? Same goes
for virtual dependencies which can be fulfilled by many different
packages.

Of course, that's just implementation-wide issue and running separate
emerge instances could result in even worse results of this problem.
But apart from that, there are side uses of multiple emerges which
would be hard to implement with your idea.

For example, I often use different PORTAGE_TMPDIR to build large
packages like xulrunner. Although it doesn't seem that hard
implementing the support for running merges in different temporary
directories at the same time, should we really waste time implementing
that? What about other possible configuration changes? What about
'USE' overrides within the environment? These ones should certainly be
respected.

So, now our daemon not only has to be able to efficiently mangle
dependency trees at runtime but also to support partial runtime
configuration changes.

The next thing is aborting merges. When running multiple emerges,
aborting one of them is as simple as pressing ^c. With daemon, we would
have to implement an ability of aborting/removing packages in runtime
-- and that would be another example of dependency tree mangling.

But in fact, implementing (at least limited) the support of such
mangling would be useful in some cases. With '--keep-going', it would
allow to drop the packages and continue merging the remaining ones
without waiting for current builds to complete.

And looking from the other side, the support for removing single
packages from the running emerge dependency tree would be useful too
with current emerge concept. Even if it just worked like
'--keep-going', it would be still better than the current workaround --
waiting for particular emerge to start and removing its ${WORKDIR} to
force failure.

--
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>
 
Old 04-14-2010, 06:10 PM
Brian Harring
 
Default RESTRICT=parallel for builds that can't be executed in parallel)

On Wed, Apr 14, 2010 at 04:20:18PM +0200, Michaaa GGGrny wrote:
> On Tue, 13 Apr 2010 23:10:16 -0700
> Brian Harring <ferringb@gmail.com> wrote:
>
> > Running multiple emerges in parallel is already a bad idea. The
> > solution for that case is for the new/second emerge to feed the
> > request into the original emerge (or a daemon).
>
> Although such solution will be useful in many cases indeed, there are
> still many advantages of having few separate emerge calls running
> in parallel.

The examples you give are fine and dandy, but if done via parallel
emerge you can run into situations where PM 1 just added pkg A as a
dep for PKG B, and PM 2 is removing pkg A due to a blocker for pkg C.

Running multiple emerge's in parallel is unsafe due to the fact
they've got two potentially very different plans as to what is being
done, and that there is no possibility to ensure that pkg D that PM-2
is building isn't affected by PM-1 building something (upgrading a
dependency of pkg D for example).

Yes you can get away with it occasionally, that doesn't mean it's
safe however.

> The next thing is aborting merges. When running multiple emerges,
> aborting one of them is as simple as pressing ^c. With daemon, we would
> have to implement an ability of aborting/removing packages in runtime
> -- and that would be another example of dependency tree mangling.

Aborting merges is a very, very bad idea. Consider a pkg that has
dlopen'd plugins, and just went through an ABI change for that
interface. If you interupt that merge it's entirely possible you'll
get just the lib merged (meaning a segfault on loadup of the plugins),
or vice versa (old lib is still in place, but new plugins are there).

Don't abort merges- that really should be effectively an atomic OP,
not interuptible.

~harring
 
Old 04-14-2010, 06:35 PM
Michał Górny
 
Default RESTRICT=parallel for builds that can't be executed in parallel)

On Wed, 14 Apr 2010 11:10:29 -0700
Brian Harring <ferringb@gmail.com> wrote:

> Running multiple emerge's in parallel is unsafe due to the fact
> they've got two potentially very different plans as to what is being
> done, and that there is no possibility to ensure that pkg D that PM-2
> is building isn't affected by PM-1 building something (upgrading a
> dependency of pkg D for example).
>
> Yes you can get away with it occasionally, that doesn't mean it's
> safe however.

I agree with you -- such operations should be performed with
appropriate caution, on user's own responsibility. But this doesn't
mean we should prevent user from being able to do that.

> Aborting merges is a very, very bad idea. Consider a pkg that has
> dlopen'd plugins, and just went through an ABI change for that
> interface. If you interupt that merge it's entirely possible you'll
> get just the lib merged (meaning a segfault on loadup of the
> plugins), or vice versa (old lib is still in place, but new plugins
> are there).
>
> Don't abort merges- that really should be effectively an atomic OP,
> not interuptible.

Sorry, by 'aborting merges' I meant rather aborting the build process
before the 'merge' phase -- or even aborting it before it is started
(i.e. before it unpacks a load of files into ${PORTAGE_TMPDIR}
without sufficient space).

--
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>
 
Old 04-14-2010, 06:47 PM
Duncan
 
Default RESTRICT=parallel for builds that can't be executed in parallel)

Brian Harring posted on Wed, 14 Apr 2010 11:10:29 -0700 as excerpted:

>> The next thing is aborting merges. When running multiple emerges,
>> aborting one of them is as simple as pressing ^c. With daemon, we would
>> have to implement an ability of aborting/removing packages in runtime
>> -- and that would be another example of dependency tree mangling.
>
> Aborting merges is a very, very bad idea. Consider a pkg that has
> dlopen'd plugins, and just went through an ABI change for that
> interface. If you interupt that merge it's entirely possible you'll get
> just the lib merged (meaning a segfault on loadup of the plugins), or
> vice versa (old lib is still in place, but new plugins are there).
>
> Don't abort merges- that really should be effectively an atomic OP, not
> interuptible.

Umm... I think you two are using the same words for different things.

Definitely, aborting qmerge (transfer to the live filesystem) isn't a good
idea, but in context, it's plain that MG's talking about the entire merge
process, from setup, unpack and prepare thru qmerge and postinst (which is
how the terms are used in the ebuild qmerge vs. ebuild merge context as
well). Clearly the entire merge doesn't need to be atomic, or we'd not be
talking about parallel merges in the first place, nor would we have
available all the individual ebuild <command> substeps.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 

Thread Tools




All times are GMT. The time now is 12:43 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org