FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-09-2009, 06:46 PM
Chuan-kai Lin
 
Default #518696 ITP: parallel -- build and execute command lines from standard input in parallel]

On Mon, Mar 09, 2009 at 11:40:51AM +0100, Samuel Thibault wrote:
> A lot of applications (including md5sum) would not necessarily print
> their output atomically and then you get mixed output. Either we add
> the option to findutils, or we package parallel.

It appears to me that you can get the same functionality by using xargs
with an adapted version of annotate-output(1) which is a part of
devscripts. Are there other reasons to use parallel?

--
Chuan-kai Lin
http://web.cecs.pdx.edu/~cklin/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-09-2009, 09:35 PM
Cyril Brulebois
 
Default #518696 ITP: parallel -- build and execute command lines from standard input in parallel]

Samuel Thibault <samuel.thibault@ens-lyon.org> (09/03/2009):
> which makes parallel not take a command, but executes commands from
> stdin. That can however be obtained by xargs sh -c. Another option
> that xargs misses is
>
> -j +N Add N to the number of CPUs. Run this many jobs in parallel.
> For compute intensive jobs -j +0 is useful as it will run
> number-of-cpus jobs in parallel.
>
> -j -N Subtract N from the number of CPUs. Run this many jobs in
> parallel. If the evaluated number is less than 1 then 1 will
> be used.
>
> -j N% Multiply N% with the number of CPUs. Run this many jobs in
> parallel. If the evaluated number is less than 1 then 1 will
> be used.

Particularly useful in cluster environments.

Mraw,
KiBi.
 
Old 03-09-2009, 10:19 PM
Chuan-kai Lin
 
Default #518696 ITP: parallel -- build and execute command lines from standard input in parallel]

On Mon, Mar 09, 2009 at 10:57:57PM +0100, Samuel Thibault wrote:
> I thought at first "it's not particularly convenient", then "well, so
> what". Now I'm thinking "Mmm, but people won't know they should do it
> and blame xargs for being broken". Also annotate-output is not enough
> when programs e.g. output Packages entries, which not only should be
> line-atomic, but also paragraph-atomic...

Below is what I had in mind when I mentioned adapting annotate-output to
a different "atomic-output" script. This script is usefull not just
with "xargs -P", but also with "make -j" and with standard background
jobs (shell & operator), all of which produce mixed output.

Similarly, about matching the number of parallel jobs with the number of
processors/cores, we can write a script "ncpus" which returns the number
of processors/cores/hyper-threads. You can use the ncpus script with
xargs, with make, or with my new project mdm (mdm.berlios.de)...

I consider separating these concerns (output management, processor
thread detection) into small, separate, and reusable scripts a cleaner
solution. Of course, doing it this way requires some user education, so
a few manpage updates (for example, adding atomic-output and ncpus to
the SEE ALSO section of xargs) may be in order.

----------

#! /bin/bash
# Display stdout and stderr output after program termination
# Adapted from annotate-output by Chuan-kai Lin
# Original annotate-output author info and copyright notice as follows

# this script was downloaded from:
# http://jeroen.a-eskwadraat.nl/sw/annotate
# and is part of devscripts 2.10.46

# Executes a program annotating the output linewise with time and stream
# Version 1.2

# Copyright 2003, 2004 Jeroen van Wolffelaar <jeroen@wolffelaar.nl>

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

OUT=`mktemp /tmp/atomic.XXXXXX` || exit 1
ERR=`mktemp /tmp/atomic.XXXXXX` || exit 1

echo "------ `date +%H:%M:%S` Started $@" > $ERR
echo "------ STDERR" >> $ERR
echo "------ STDOUT" >> $OUT
"$@" >> $OUT 2>> $ERR ; EXIT=$?

cat $ERR
cat $OUT
echo "------ `date +%H:%M:%S` Finished with exitcode $EXIT"
rm -f $OUT $ERR

exit $EXIT

--
Chuan-kai Lin
http://web.cecs.pdx.edu/~cklin/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 04:11 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org