FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Portage Developer

 
 
LinkBack Thread Tools
 
Old 05-14-2012, 08:10 PM
Mike Frysinger
 
Default prepstrip/ecompressdir: parallelize operations

On Monday 14 May 2012 15:08:32 Zac Medico wrote:
> Actually, the inode_var_name thing will not work unless it's all in
> one process.

hmm, true, but that's the level we currently parallelize at, so it's fine. we
do one subprocess per ELF and that includes the strip/splitdebug/splitsources.

parallelizing more than on a per-ELF basis will require much finer grained
queues which, while possible, would make the file much harder to hack on and
extend. and i'm not sure we'd see that much of a gain.
-mike
 
Old 05-14-2012, 10:42 PM
Zac Medico
 
Default prepstrip/ecompressdir: parallelize operations

On 05/14/2012 01:10 PM, Mike Frysinger wrote:
> On Monday 14 May 2012 15:08:32 Zac Medico wrote:
>> Actually, the inode_var_name thing will not work unless it's all in
>> one process.
>
> hmm, true, but that's the level we currently parallelize at, so it's fine. we
> do one subprocess per ELF and that includes the strip/splitdebug/splitsources.
>
> parallelizing more than on a per-ELF basis will require much finer grained
> queues which, while possible, would make the file much harder to hack on and
> extend. and i'm not sure we'd see that much of a gain.
> -mike

The thing is, in the case of hardlinks, we're parallelizing multiple
times on the *same* elf. Anyway, I've fixed it by using a directory full
of hardlinks, in these commits:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=ad944275b88a50d2a1f694826b1 27cceb9221e78
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9ed00a9e70a3705164a5349145f f467e5c40ddfd
--
Thanks,
Zac
 
Old 05-14-2012, 11:13 PM
Mike Frysinger
 
Default prepstrip/ecompressdir: parallelize operations

On Monday 14 May 2012 18:42:07 Zac Medico wrote:
> On 05/14/2012 01:10 PM, Mike Frysinger wrote:
> > On Monday 14 May 2012 15:08:32 Zac Medico wrote:
> >> Actually, the inode_var_name thing will not work unless it's all in
> >> one process.
> >
> > hmm, true, but that's the level we currently parallelize at, so it's
> > fine. we do one subprocess per ELF and that includes the
> > strip/splitdebug/splitsources.
> >
> > parallelizing more than on a per-ELF basis will require much finer
> > grained queues which, while possible, would make the file much harder to
> > hack on and extend. and i'm not sure we'd see that much of a gain.
>
> The thing is, in the case of hardlinks, we're parallelizing multiple
> times on the *same* elf. Anyway, I've fixed it by using a directory full
> of hardlinks, in these commits:

well, realistically speaking, hardlinking has been broken before the
parallelization work.
https://bugs.gentoo.org/400767

> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=ad9442
> 75b88a50d2a1f694826b127cceb9221e78
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9ed00
> a9e70a3705164a5349145ff467e5c40ddfd

i'll go through it and see what's what
-mike
 

Thread Tools




All times are GMT. The time now is 07:21 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org