thread issue

On 8/4/2011 9:40 AM, lina wrote:
> On Thu, Aug 4, 2011 at 10:29 PM, Robert Baron
> <robertbartlettbaron@gmail.com> wrote:
>> have you tried adding an '&' to the tasks you think can be run in
>> parallel (as in running them in the background (ie 'mycmd myargs &'))?
>
> Thanks,
>
> "&" is cool.
> now is fully running, but, there is another thing slow it down, the
> nice level is 19. why is it so high? are there some root's setting?

Nice levels can be deceiving to those who don't understand how kernel
scheduling works. For example, it is possible, and happens all the time
really, that a program with a nice level of 19 will fully consume 100%
of a processor's time until the program completes.

The only time nice comes into play is when scheduling contention exists.
This only occurs when there are more ready-to-run processes/threads
than available processor time slice slots. On an 8 processor desktop
with the default 1000Hz scheduler you have 8,000 scheduling slices per
second. So unless your system has 8,001 ready to run processes every
second, that nice level 19 process will get all the CPU time it needs.

Nice is a relic of early UNIX when businesses and universities would
have dozens to hundreds of concurrent users on serial TTY terminals
running programs on a shared single processor machine. Nice allowed
administrators to provide "fair" access to all users, or to make sure a
critical batch job got the cycles it needed, even with many users
running their programs. On today's multi-core desktops, nice is
literally irrelevant. This is actually the case for most business
UNIX/Linux systems as well, as almost all of them have excess processor
capacity for their actual workloads.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E3ABA81.1070902@hardwarefreak.com">http://lists.debian.org/4E3ABA81.1070902@hardwarefreak.com

On Thu, Aug 04, 2011 at 10:23:08PM +0800, lina wrote:
> Thanks for suggestions,
>
> Actually I got a job which contains several small jobs inside.
>
> if run the bash script, it will do those one by one and it is pretty
> slow, waiting ...
>
> I can run each small jobs separately, but use a bash script kind of
> easy to make some changes in amount and manage.
>
> I just wonder are there some simple way to do it?

What you could try is converting your bash script to a makefile, you can
then let make do the work of parallelizing your jobs with "make -j 8
-f yourmakefile".

--
Paul Saunders

On Thu, Aug 04, 2011 at 10:28:01AM -0500, Stan Hoeppner wrote:
> On 8/4/2011 9:40 AM, lina wrote:
> > On Thu, Aug 4, 2011 at 10:29 PM, Robert Baron
> > <robertbartlettbaron@gmail.com> wrote:
> >> have you tried adding an '&' to the tasks you think can be run in
> >> parallel (as in running them in the background (ie 'mycmd myargs &'))?
> >
> > Thanks,
> >
> > "&" is cool.
> > now is fully running, but, there is another thing slow it down, the
> > nice level is 19. why is it so high? are there some root's setting?
>
> Nice levels can be deceiving to those who don't understand how kernel
> scheduling works. For example, it is possible, and happens all the time
> really, that a program with a nice level of 19 will fully consume 100%
> of a processor's time until the program completes.
>
> The only time nice comes into play is when scheduling contention exists.
> This only occurs when there are more ready-to-run processes/threads
> than available processor time slice slots. On an 8 processor desktop
> with the default 1000Hz scheduler you have 8,000 scheduling slices per
> second. So unless your system has 8,001 ready to run processes every
> second, that nice level 19 process will get all the CPU time it needs.
>
> Nice is a relic of early UNIX when businesses and universities would
> have dozens to hundreds of concurrent users on serial TTY terminals
> running programs on a shared single processor machine. Nice allowed
> administrators to provide "fair" access to all users, or to make sure a
> critical batch job got the cycles it needed, even with many users
> running their programs. On today's multi-core desktops, nice is
> literally irrelevant. This is actually the case for most business
> UNIX/Linux systems as well, as almost all of them have excess processor
> capacity for their actual workloads.

I wouldn't say that nice is completely irrelevant. I run BOINC on my
desktop machine and rely on the BOINC tasks having more nice (if I can
take the opportunity to butcher English) than other tasks such as the
web browser, the window manager and so on.

Doing so also allows me to tell my CPU scheduler to ignore niced
processes and only spin up the CPU to full speed when an important task
wants it (as opposed to always running at full speed doing the BOINC
calculations).

--
Paul Saunders

On Thu, 04 Aug 2011 16:28:41 +0100, Darac Marjal wrote:

> On Thu, Aug 04, 2011 at 10:23:08PM +0800, lina wrote:
>> Thanks for suggestions,
>>
>> Actually I got a job which contains several small jobs inside.
>>
>> if run the bash script, it will do those one by one and it is pretty
>> slow, waiting ...
>>
>> I can run each small jobs separately, but use a bash script kind of
>> easy to make some changes in amount and manage.
>>
>> I just wonder are there some simple way to do it?
>
> What you could try is converting your bash script to a makefile, you can
> then let make do the work of parallelizing your jobs with "make -j 8 -f
> yourmakefile".

That's a clever use of make. Wish I'd thought of that!



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: j1ee5l$h6s$1@dough.gmane.org">http://lists.debian.org/j1ee5l$h6s$1@dough.gmane.org

On Thu, Aug 4, 2011 at 11:28 PM, Darac Marjal <mailinglist@darac.org.uk> wrote:
> On Thu, Aug 04, 2011 at 10:23:08PM +0800, lina wrote:
>> Thanks for suggestions,
>>
>> Actually I got a job which contains several small jobs inside.
>>
>> if run the bash script, it will do those one by one and it is pretty
>> slow, waiting ...
>>
>> I can run each small jobs separately, but use a bash script kind of
>> easy to make some changes in amount and manage.
>>
>> I just wonder are there some simple way to do it?
>
> What you could try is converting your bash script to a makefile, you can
> then let make do the work of parallelizing your jobs with "make -j 8
> -f yourmakefile".

Thanks,

how to convert a bash script to a makefile?

>
> --
> Paul Saunders
>



--
Best Regards,

lina


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAG9cJmkBJCB5BTRzoL-cwo-vzOURm9XXKRAfNafbNwM7p8OqPw@mail.gmail.com">http://lists.debian.org/CAG9cJmkBJCB5BTRzoL-cwo-vzOURm9XXKRAfNafbNwM7p8OqPw@mail.gmail.com

On Thu, Aug 4, 2011 at 11:28 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 8/4/2011 9:40 AM, lina wrote:
>> On Thu, Aug 4, 2011 at 10:29 PM, Robert Baron
>> <robertbartlettbaron@gmail.com> wrote:
>>> have you tried adding an '&' to the tasks you think can be run in
>>> parallel (as in running them in the background (ie 'mycmd myargs &'))?
>>
>> Thanks,
>>
>> "&" is cool.
>> now is fully running, but, there is another thing slow it down, the
>> nice level is 19. why is it so high? are there some root's setting?
>
> Nice levels can be deceiving to those who don't understand how kernel
> scheduling works. *For example, it is possible, and happens all the time
> really, that a program with a nice level of 19 will fully consume 100%
> of a processor's time until the program completes.
>
> The only time nice comes into play is when scheduling contention exists.
> *This only occurs when there are more ready-to-run processes/threads
> than available processor time slice slots. *On an 8 processor desktop
> with the default 1000Hz scheduler you have 8,000 scheduling slices per
> second. *So unless your system has 8,001 ready to run processes every
> second, that nice level 19 process will get all the CPU time it needs.
>
> Nice is a relic of early UNIX when businesses and universities would
> have dozens to hundreds of concurrent users on serial TTY terminals
> running programs on a shared single processor machine. *Nice allowed
> administrators to provide "fair" access to all users, or to make sure a
> critical batch job got the cycles it needed, even with many users
> running their programs. *On today's multi-core desktops, nice is
> literally irrelevant. *This is actually the case for most business
> UNIX/Linux systems as well, as almost all of them have excess processor
> capacity for their actual workloads.

Actually the nice-concern was in cluster.

I noticed only run .sh a bit heavy staff, the nice level is 19. for
the TTY terminals is 0.

is it possible to deceive the .sh job as TTY terminals. (I should take
blame here cause thinking like this, but ...)

I can't use qsub or mpi here, the thread is the only option I got.

Thanks,
>
> --
> Stan
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/4E3ABA81.1070902@hardwarefreak.com
>
>



--
Best Regards,

lina


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAG9cJmm134pug9oaAkFprLAVOmvsM3ECHM279Q2zdUK13-q+PA@mail.gmail.com">http://lists.debian.org/CAG9cJmm134pug9oaAkFprLAVOmvsM3ECHM279Q2zdUK13-q+PA@mail.gmail.com

On Thu, Aug 04, 2011 at 11:37:57PM +0800, lina wrote:
> On Thu, Aug 4, 2011 at 11:28 PM, Darac Marjal <mailinglist@darac.org.uk> wrote:
> > On Thu, Aug 04, 2011 at 10:23:08PM +0800, lina wrote:
> >> Thanks for suggestions,
> >>
> >> Actually I got a job which contains several small jobs inside.
> >>
> >> if run the bash script, it will do those one by one and it is pretty
> >> slow, waiting ...
> >>
> >> I can run each small jobs separately, but use a bash script kind of
> >> easy to make some changes in amount and manage.
> >>
> >> I just wonder are there some simple way to do it?
> >
> > What you could try is converting your bash script to a makefile, you can
> > then let make do the work of parallelizing your jobs with "make -j 8
> > -f yourmakefile".
>
> Thanks,
>
> how to convert a bash script to a makefile?

http://lmddgtfy.com/?q=makefile+howto

--
Paul Saunders

On 8/4/2011 10:33 AM, Darac Marjal wrote:
> On Thu, Aug 04, 2011 at 10:28:01AM -0500, Stan Hoeppner wrote:
>> On 8/4/2011 9:40 AM, lina wrote:
>>> On Thu, Aug 4, 2011 at 10:29 PM, Robert Baron
>>> <robertbartlettbaron@gmail.com> wrote:
>>>> have you tried adding an '&' to the tasks you think can be run in
>>>> parallel (as in running them in the background (ie 'mycmd myargs &'))?
>>>
>>> Thanks,
>>>
>>> "&" is cool.
>>> now is fully running, but, there is another thing slow it down, the
>>> nice level is 19. why is it so high? are there some root's setting?
>>
>> Nice levels can be deceiving to those who don't understand how kernel
>> scheduling works. For example, it is possible, and happens all the time
>> really, that a program with a nice level of 19 will fully consume 100%
>> of a processor's time until the program completes.
>>
>> The only time nice comes into play is when scheduling contention exists.
>> This only occurs when there are more ready-to-run processes/threads
>> than available processor time slice slots. On an 8 processor desktop
>> with the default 1000Hz scheduler you have 8,000 scheduling slices per
>> second. So unless your system has 8,001 ready to run processes every
>> second, that nice level 19 process will get all the CPU time it needs.
>>
>> Nice is a relic of early UNIX when businesses and universities would
>> have dozens to hundreds of concurrent users on serial TTY terminals
>> running programs on a shared single processor machine. Nice allowed
>> administrators to provide "fair" access to all users, or to make sure a
>> critical batch job got the cycles it needed, even with many users
>> running their programs. On today's multi-core desktops, nice is
>> literally irrelevant. This is actually the case for most business
>> UNIX/Linux systems as well, as almost all of them have excess processor
>> capacity for their actual workloads.
>
> I wouldn't say that nice is completely irrelevant. I run BOINC on my
> desktop machine and rely on the BOINC tasks having more nice (if I can
> take the opportunity to butcher English) than other tasks such as the
> web browser, the window manager and so on.
>
> Doing so also allows me to tell my CPU scheduler to ignore niced
> processes and only spin up the CPU to full speed when an important task
> wants it (as opposed to always running at full speed doing the BOINC
> calculations).

Thanks for pointing out the exception to the rule. I would think
affinity (taskset) is likely more popular with BOINC users than nice,
though. I guess you simply don't want an idle core, ever. Must be hard
on your electric bill.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E3AC117.2050904@hardwarefreak.com">http://lists.debian.org/4E3AC117.2050904@hardwarefreak.com

On 8/4/2011 10:34 AM, Walter Hurry wrote:
> On Thu, 04 Aug 2011 16:28:41 +0100, Darac Marjal wrote:
>
>> On Thu, Aug 04, 2011 at 10:23:08PM +0800, lina wrote:
>>> Thanks for suggestions,
>>>
>>> Actually I got a job which contains several small jobs inside.
>>>
>>> if run the bash script, it will do those one by one and it is pretty
>>> slow, waiting ...
>>>
>>> I can run each small jobs separately, but use a bash script kind of
>>> easy to make some changes in amount and manage.
>>>
>>> I just wonder are there some simple way to do it?
>>
>> What you could try is converting your bash script to a makefile, you can
>> then let make do the work of parallelizing your jobs with "make -j 8 -f
>> yourmakefile".
>
> That's a clever use of make. Wish I'd thought of that!

Simply backgrounding the processes in the existing shell script is
easier. Using the make file option is simply extra work with no gain.
I do know of other cases where the make file option is better. Just not
this case.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E3AC1FE.70904@hardwarefreak.com">http://lists.debian.org/4E3AC1FE.70904@hardwarefreak.com

Hello List:

just use a job scheduler as SLURM:

http://packages.debian.org/wheezy/slurm-llnl

Jerome

On 04/08/11 16:23, lina wrote:

Thanks for suggestions,

Actually I got a job which contains several small jobs inside.

if run the bash script, it will do those one by one and it is pretty
slow, waiting ...

I can run each small jobs separately, but use a bash script kind of
easy to make some changes in amount and manage.

I just wonder are there some simple way to do it?

another question, if I don't have root previlege, can I adjust my nice
level in some cluster? I noticed mine NI was kind of 19, totally crazy
slow.

On Thu, Aug 4, 2011 at 10:15 PM, Frank Lanitz<frank@frank.uvena.de> wrote:

Am 04.08.2011 15:12, schrieb lina:

Hi,

I noticed when make -j 8, the 8 cores can be fully occupied.

can I use some way to enable 8 cores at the same time when I run
something, such as a bash script?


Hard to say as it depends on the software you are running. On shell
scripts you might can do it by intelligent forking of processes doing
the single tasks.


How can I fork of processes doing each single tasks?


Other applications do need to support multi threading in most cases
already inside source code.

Cheers,
Frank


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/4E3AA979.7010406@frank.uvena.de









--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4E3AC2BA.7090109@rezozer.net">http://lists.debian.org/4E3AC2BA.7090109@rezozer.net