This is not Fedora specific, so apologies for posting it here. I used to
post this kind of questions in comp.linux.misc or the like but I don't
have access to usenet groups anymore.

So here is the question. Suppose I have several processes that run
concurrently and each outputs stuff to stdout. Can the combined output be
intermingled?


Example: script ioTest.sh:


#!/bin/bash

i=0
while [ $i -lt 100 ];
do
echo "AAAAA" & # e.g. 500 As
echo "BBBBB" &
echo "CCCCC" &

i=$(($i+1))
done


Then

./ioTest.sh > out

The A's and B's and C's always seem to be in contiguous blocks of 500 in
the output file, but is that guaranteed?

Thanks!

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 04/14/2012 09:35 AM, Amadeus W.M. wrote:
> This is not Fedora specific, so apologies for posting it here. I used to
> post this kind of questions in comp.linux.misc or the like but I don't
> have access to usenet groups anymore.
>
> So here is the question. Suppose I have several processes that run
> concurrently and each outputs stuff to stdout. Can the combined output be
> intermingled?
>
>
> Example: script ioTest.sh:
>
>
> #!/bin/bash
>
> i=0
> while [ $i -lt 100 ];
> do
> echo "AAAAA" & # e.g. 500 As
> echo "BBBBB" &
> echo "CCCCC" &
>
> i=$(($i+1))
> done
>
>
> Then
>
> ./ioTest.sh > out
>
> The A's and B's and C's always seem to be in contiguous blocks of 500 in
> the output file, but is that guaranteed?
>
> Thanks!
>

No.... Not only that, you are placing the echo commands in the background. So, it
is certainly possible that the script will finish before the echos are completed.
Not only that, there is no guarantee that all output from the echos will be written
to "out". You can see this if you put a sleep after the last echo.

[egreshko@meimei test]$ grep ^A out | wc
97 97 582
[egreshko@meimei test]$ grep ^B out | wc
94 94 564
[egreshko@meimei test]$ grep ^C out | wc
96 96 576

Your results will vary depending on the number of CPU's. If you have a single CPU
system, you may very well get all 300.

--
Never be afraid to laugh at yourself, after all, you could be missing out on the joke
of the century. -- Dame Edna Everage
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 04/14/2012 10:33 AM, Ed Greshko wrote:
> No.... Not only that, you are placing the echo commands in the background. So, it
> is certainly possible that the script will finish before the echos are completed.
> Not only that, there is no guarantee that all output from the echos will be written
> to "out". You can see this if you put a sleep after the last echo.
>
> [egreshko@meimei test]$ grep ^A out | wc
> 97 97 582
> [egreshko@meimei test]$ grep ^B out | wc
> 94 94 564
> [egreshko@meimei test]$ grep ^C out | wc
> 96 96 576
>
> Your results will vary depending on the number of CPU's. If you have a single CPU
> system, you may very well get all 300.

My "poor" English may be misleading/ambiguous .... The output of the greps was
*without* the sleep in the script.

--
Never be afraid to laugh at yourself, after all, you could be missing out on the joke
of the century. -- Dame Edna Everage
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

>
> [egreshko@meimei test]$ grep ^A out | wc
> 97 97 582
> [egreshko@meimei test]$ grep ^B out | wc
> 94 94 564
> [egreshko@meimei test]$ grep ^C out | wc
> 96 96 576
>


I replicated this and indeed I don't get 100 lines of As, Bs and Cs.
That's a new problem. Why don't I get all? Shouldn't all 300 echos have
been launched when the script completes? And if the script has completed
and there are echos running in the background, they should complete, no?

I put a "wait" at the end of the script to wait for the completion of the
echos, but I still get fewer lines than 100 per A, B, C.

This is aggravating. So not all echos get executed (why?), yet I still
don't see the output mingled.


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On Sat, Apr 14, 2012 at 01:35:38AM +0000, Amadeus W.M. wrote:
> So here is the question. Suppose I have several processes that run
> concurrently and each outputs stuff to stdout. Can the combined output be
> intermingled?

If you just send the output to a file, you've no way of knowing exactly
when it will be output, or whether it will be buffered before writing.
You can mitigate this a bit by making every output line have a sequence and
process identifier--the PID and date in seconds would work--so you could
separate the streams later.

If you really would like to get output in sequence, write to a pipe, and
have a reader process drain the pipe to a logfile. It's pretty easy; look
at "mknod" with the 'p' option, or "mkfifo". I'd still suggest tagging
each output line with an identifier and sequence number.

Cheers,
--
Dave Ihnat
dihnat@dminet.com
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 04/14/2012 09:33 PM, Dave Ihnat wrote:
> On Sat, Apr 14, 2012 at 01:35:38AM +0000, Amadeus W.M. wrote:
>> So here is the question. Suppose I have several processes that run
>> concurrently and each outputs stuff to stdout. Can the combined output be
>> intermingled?
> If you just send the output to a file, you've no way of knowing exactly
> when it will be output, or whether it will be buffered before writing.
> You can mitigate this a bit by making every output line have a sequence and
> process identifier--the PID and date in seconds would work--so you could
> separate the streams later.
>
> If you really would like to get output in sequence, write to a pipe, and
> have a reader process drain the pipe to a logfile. It's pretty easy; look
> at "mknod" with the 'p' option, or "mkfifo". I'd still suggest tagging
> each output line with an identifier and sequence number.
>

Yes, the pipe idea would work well and ensure you get all the output.

The one problem would be if, as the OP's script is written, backgrounding of
processes is done there is no way to control the order of data being written to the
pipe. And, as you pointed out, you may want to write out sequence numbers if that is
important.

--
Never be afraid to laugh at yourself, after all, you could be missing out on the joke
of the century. -- Dame Edna Everage
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On Sat, Apr 14, 2012 at 09:49:00PM +0800, Ed Greshko wrote:
> The one problem would be if, as the OP's script is written,
> backgrounding of processes is done there is no way to control the
> order of data being written to the pipe. And, as you pointed out,
> you may want to write out sequence numbers if that is important.

Well, it's guaranteed it will be in the order written by each process,
since the write to the pipe is queued in request order. I'm
anal-retentive; the more useful data, the better. The identifier &
sequence number gives me more data I can mung, and strip if I don't
need it.

Cheers,
--
Dave Ihnat
dihnat@dminet.com
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 04/14/2012 10:09 PM, Dave Ihnat wrote:
> On Sat, Apr 14, 2012 at 09:49:00PM +0800, Ed Greshko wrote:
>> The one problem would be if, as the OP's script is written,
>> backgrounding of processes is done there is no way to control the
>> order of data being written to the pipe. And, as you pointed out,
>> you may want to write out sequence numbers if that is important.
> Well, it's guaranteed it will be in the order written by each process,
> since the write to the pipe is queued in request order. I'm
> anal-retentive; the more useful data, the better. The identifier &
> sequence number gives me more data I can mung, and strip if I don't
> need it.

What I was referring to was the script as written by the OP and which may he executed as

./io.sh > pipe

The script contains multiple "echo &" commands which are not guaranteed to be
executed in order, especially on a multi-CPU system.

--
Never be afraid to laugh at yourself, after all, you could be missing out on the joke
of the century. -- Dame Edna Everage
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On Fri, Apr 13, 2012 at 10:31 PM, Amadeus W.M. <amadeus84@verizon.net> wrote:
>>
>> [egreshko@meimei test]$ grep ^A out | wc
>> * * *97 * * *97 * * 582
>> [egreshko@meimei test]$ grep ^B out | wc
>> * * *94 * * *94 * * 564
>> [egreshko@meimei test]$ grep ^C out | wc
>> * * *96 * * *96 * * 576
>
> I replicated this and indeed I don't get 100 lines of As, Bs and Cs.
> That's a new problem. Why don't I get all? Shouldn't all 300 echos have
> been launched when the script completes? And if the script has completed
> and there are echos running in the background, they should complete, no?
>
> I put a "wait" at the end of the script to wait for the completion of the
> echos, but I still get fewer lines than 100 per A, B, C.
>
> This is aggravating. So not all echos get executed (why?), yet I still
> don't see the output mingled.

I admit I am not really clear on what output you want to get but does
this give the output you want?

./ioTest.sh | cat > out

John
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

> If you really would like to get output in sequence, write to a pipe, and
> have a reader process drain the pipe to a logfile. It's pretty easy;
> look at "mknod" with the 'p' option, or "mkfifo". I'd still suggest
> tagging each output line with an identifier and sequence number.
>

For the sake of the argument, assume I echo 500 As, 500 Bs and 500 Cs.

I don't care which process the output is coming from. It doesn't matter
which order the As, Bs and Cs are output. All I care about is that I
don't get 349As followed by 245Bs, etc. I want to see blocks of 500 each.

I don't see how echoing into a pipe would change the problem.
Theoretically, if several processes (e.g. echo) are running in the
background, e.g. on a round robin basis, then potentially I could see
random sequences of As, Bs and Cs. It doesn't seem to be the case in
practice though. So which is it?

This has to do with the operating system internals, it's not a trivial
question.

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org