Is it legal for ebuilds to call has_version and friends in parallel? Is
it legal for ebuilds to call has_version and friends after the ebuild
process has terminated? Discuss.

--
Ciaran McCreesh

Ciaran McCreesh kirjoitti:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.
>

Do you/anybody know if they are used in parallel in the tree at the moment?

Regards,
Petteri

On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> Is it legal for ebuilds to call has_version and friends in parallel? Is
> it legal for ebuilds to call has_version and friends after the ebuild
> process has terminated? Discuss.

If the pm implements read/write locking on the underlying datastore
(which it should probably have regardless of this request) then I
don't see a problem in parallel has_version calls.

I don't get your second example..do you mean the ebuild is running
has_version in the background and then terminating?

>
> --
> Ciaran McCreesh
>
>
--
gentoo-dev@gentoo.org mailing list

On Mon, 31 Dec 2007 05:03:21 +0200
Petteri Räty <betelgeuse@gentoo.org> wrote:
> Ciaran McCreesh kirjoitti:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
> >
>
> Do you/anybody know if they are used in parallel in the tree at the
> moment?

I can't see anything obvious, but that doesn't mean there aren't a few
weird hiding cases.

--
Ciaran McCreesh

On Sun, 30 Dec 2007 20:11:16 -0800
"Alec Warner" <antarus@gentoo.org> wrote:
> On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > Is it legal for ebuilds to call has_version and friends in
> > parallel? Is it legal for ebuilds to call has_version and friends
> > after the ebuild process has terminated? Discuss.
>
> If the pm implements read/write locking on the underlying datastore
> (which it should probably have regardless of this request) then I
> don't see a problem in parallel has_version calls.

Actually, it's the communication channel that's the issue... If, for
example, has_version is implemented in terms of a request on a pipe
rather than execing a new package manager, we get into messy bash
locking territory...

> I don't get your second example..do you mean the ebuild is running
> has_version in the background and then terminating?

Yeah. Again, consider the pipe example. If the package manager closes
off the pipe when it thinks the ebuild's done, calling has_version will
get the backgrounded process SIGPIPEd.

--
Ciaran McCreesh

On Mon, Dec 31, 2007 at 02:28:44PM +0000, Ciaran McCreesh wrote:
> On Sun, 30 Dec 2007 20:11:16 -0800
> "Alec Warner" <antarus@gentoo.org> wrote:
> > On 12/30/07, Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> > > Is it legal for ebuilds to call has_version and friends in
> > > parallel? Is it legal for ebuilds to call has_version and friends
> > > after the ebuild process has terminated? Discuss.
> >
> > If the pm implements read/write locking on the underlying datastore
> > (which it should probably have regardless of this request) then I
> > don't see a problem in parallel has_version calls.
>
> Actually, it's the communication channel that's the issue... If, for
> example, has_version is implemented in terms of a request on a pipe
> rather than execing a new package manager, we get into messy bash
> locking territory...
>
> > I don't get your second example..do you mean the ebuild is running
> > has_version in the background and then terminating?
>
> Yeah. Again, consider the pipe example. If the package manager closes
> off the pipe when it thinks the ebuild's done, calling has_version will
> get the backgrounded process SIGPIPEd.

Depends on the implementation; for pkgcore, if that comm pipe is
dead, the ebuild env *should* be dead, or dieing. Background'ing
processes from that env isn't valid imo, either.

If you're refering to an ebuild that parallelizes itself while
executing, iow, parallelization w/in the ebuild env/phase execution,
I'd look more at being able to batch commands instead of trying to run
them in parallel. Reasoning follows-

1) if doing an exec approach to service the request, this means
reparsing of involved files for each request- inefficient, potentially
horribly so on crappy hardware/setups.
2) screws up the pipe approach, should folks take it for control/env
introspection gains.

Summarizing, executing has_version (and friends)
concurrently has it's own issues performance wise, and implementation
wise; growing batch functionality into portageq however avoids those
issues, and would be faster- thus the route I'd advocate.

~harring

On Fri, 4 Jan 2008 18:50:56 -0800
Brian Harring <ferringb@gmail.com> wrote:
> Depends on the implementation; for pkgcore, if that comm pipe is
> dead, the ebuild env *should* be dead, or dieing. Background'ing
> processes from that env isn't valid imo, either.

Right. Paludis will give a weird die message but not actually fail if
you do:

src_compile() {
{ sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
}

> If you're refering to an ebuild that parallelizes itself while
> executing, iow, parallelization w/in the ebuild env/phase execution,
> I'd look more at being able to batch commands instead of trying to
> run them in parallel.

That's its own slippery slope. Because of limited size pipes, the
following causes allllll sorts of trouble:

pkg_setup()
{
portageq match ${ROOT} cat/some-pkg | while read a ; do
if has_version ="${a}" ; then
echo "yes to ${a}"
else
echo "no to ${a}"
fi
done
}

The problem is thus: the has_version and portageq match here can be run
in parallel by bash. The portageq match output can be longer than the
maximum size of a pipe. Thus, if the above is legal, no lock that is
visible to the has_version can be held by portageq match once it starts
producing output.

--
Ciaran McCreesh

Ciaran McCreesh wrote:
> On Fri, 4 Jan 2008 18:50:56 -0800
> Brian Harring <ferringb@gmail.com> wrote:
>> Depends on the implementation; for pkgcore, if that comm pipe is
>> dead, the ebuild env *should* be dead, or dieing. Background'ing
>> processes from that env isn't valid imo, either.
>
> Right. Paludis will give a weird die message but not actually fail if
> you do:
>
> src_compile() {
> { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> }

is & allowed in ebuilds? should?

lu

--

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

--
gentoo-dev@gentoo.org mailing list

Luca Barbato kirjoitti:

Ciaran McCreesh wrote:

On Fri, 4 Jan 2008 18:50:56 -0800
Brian Harring <ferringb@gmail.com> wrote:
Depends on the implementation; for pkgcore, if that comm pipe is
dead, the ebuild env *should* be dead, or dieing. Background'ing
processes from that env isn't valid imo, either.

Right. Paludis will give a weird die message but not actually fail if
you do:

src_compile() {
{ sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
}


is & allowed in ebuilds? should?

lu



I would say that nothing started in src_* functions should be running
when the function exits.


Regards,
Petteri

On Sat, 05 Jan 2008 18:29:51 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > src_compile() {
> > { sleep 10 ; has_version '>=app-misc/foo-1.23' ; } &
> > }
>
> is & allowed in ebuilds? should?

Banning it entirely is excessive. Banning leaving any attached
processes between phases is hopefully not going to upset anyone...

--
Ciaran McCreesh