- Bash question: get output as a variable?
 

On Fri, 5 Feb 2010 10:44:28 -0500 (EST), bruno wrote:
> Ken Teague wrote:
>> On Fri, Feb 5, 2010 at 12:39 AM, bruno <bruno.debian@cyberoso.com> wrote:
>>
>>> Why not simply use the t option for content listing :
>>>
>>> tar tvf * --exclude-from $EXCLUDES
>>>
>>
>> He's already creating the archive with -v. Why process the archive a
>> 2nd time just to get a listing when it comes from stdout the 1st time?
>>
>>
>>
> because it's a simplier way to get the list into a variable
> because the script gets much clearer
> because it is not safe to rely on stderr since it is supposed to display
> errors, which it might do as well
> because he would get the list of the files that have really been
> compressed, not just the processed ones
> because the overhead may not be worth the complication of trying to do
> both things at the same time
> because he might as well never use the v option (it's not useful in my
> proposition either, and it would even reduce the overhead)
>
> because it is another way to do it and nobody had proposed it before

This is off topic from the OP's question, but one of the things that I
miss in the Linux environment that I used to use a lot in the CMS
environment is CMS Pipelines. The shell supports pipelines, but they
are *single-stream* pipelines. CMS pipelines supports *multi-stream*
pipelines. That one feature alone makes CMS Pipelines so much more
powerful than shell pipelines. I wish the shell supported multi-stream
pipelines.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

Stephen Powell writes:
> I wish the shell supported multi-stream pipelines.

I think you could fake it with tee and a fifo.
--
John Hasler


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

Stephen Powell wrote:
> On Fri, 5 Feb 2010 10:44:28 -0500 (EST), bruno wrote:
>> Ken Teague wrote:
>>> On Fri, Feb 5, 2010 at 12:39 AM, bruno <bruno.debian@cyberoso.com> wrote:
>>>
>>>> Why not simply use the t option for content listing :
>>>>
>>>> tar tvf * --exclude-from $EXCLUDES
>>>>
>>> He's already creating the archive with -v. Why process the archive a
>>> 2nd time just to get a listing when it comes from stdout the 1st time?

---"because" removed---

>
> This is off topic from the OP's question, but one of the things that I
> miss in the Linux environment that I used to use a lot in the CMS
> environment is CMS Pipelines. The shell supports pipelines, but they
> are *single-stream* pipelines. CMS pipelines supports *multi-stream*
> pipelines. That one feature alone makes CMS Pipelines so much more
> powerful than shell pipelines. I wish the shell supported multi-stream
> pipelines.
>
>

As I'm not familiar with CMS, I have no idea if the following matches or
is completely off target ...

The default pipeline in shells always operates with stdout/stdin, which
is a bit limiting.

But, you can work around it, a bit, like this:

cat abc_does_not_exist 2>&1 | wc

The 'cat' error message gets sent to wc. The normal cat output is still
to stdout, so it also would be processed by wc, when the file exists.

The shell also allows you to open your own descriptors to files, using:

exec 5>abc
date >&5
cat abc
Fri Feb 5 09:15:15 PST 2010

Or, you can set the descriptor to point to another one:

exec 5>&1
date >&5
Fri Feb 5 09:15:15 PST 2010

You can also create "named pipes" using 'mknod a_name p'. For example:

mknod fifo1 p
mknod fifo2 p
tr '[A-Z]' '[a-z]' < fifo1 >fifo2 &
[1] 25269
echo ABCDEFG > fifo1
cat fifo2
abcdefg
[1]+ Done tr '[A-Z]' '[a-z]' <fifo1 >fifo2

Perhaps one or another of the above will do what you want?

--
Bob McGowan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

On Fri, 5 Feb 2010 11:39:26 -0500 (EST), John Hasler wrote:
> Stephen Powell writes:
>> I wish the shell supported multi-stream pipelines.
>
> I think you could fake it with tee and a fifo.

Well, I know about tee; but, although I've heard the term fifo,
I know nothing about it in a Linux/Unix/shell context. I'll look
into it. Thanks.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

On Fri, 5 Feb 2010 12:42:47 -0500 (EST), Bob McGowan wrote:
> Stephen Powell wrote:
>> This is off topic from the OP's question, but one of the things that I
>> miss in the Linux environment that I used to use a lot in the CMS
>> environment is CMS Pipelines. The shell supports pipelines, but they
>> are *single-stream* pipelines. CMS pipelines supports *multi-stream*
>> pipelines. That one feature alone makes CMS Pipelines so much more
>> powerful than shell pipelines. I wish the shell supported multi-stream
>> pipelines.
>
> As I'm not familiar with CMS, I have no idea if the following matches or
> is completely off target ...
>
> The default pipeline in shells always operates with stdout/stdin, which
> is a bit limiting.
>
> But, you can work around it, a bit, like this:
>
> cat abc_does_not_exist 2>&1 | wc
>
> The 'cat' error message gets sent to wc. The normal cat output is still
> to stdout, so it also would be processed by wc, when the file exists.
>
> The shell also allows you to open your own descriptors to files, using:
>
> exec 5>abc
> date >&5
> cat abc
> Fri Feb 5 09:15:15 PST 2010
>
> Or, you can set the descriptor to point to another one:
>
> exec 5>&1
> date >&5
> Fri Feb 5 09:15:15 PST 2010
>
> You can also create "named pipes" using 'mknod a_name p'. For example:
>
> mknod fifo1 p
> mknod fifo2 p
> tr '[A-Z]' '[a-z]' < fifo1 >fifo2 &
> [1] 25269
> echo ABCDEFG > fifo1
> cat fifo2
> abcdefg
> [1]+ Done tr '[A-Z]' '[a-z]' <fifo1 >fifo2
>
> Perhaps one or another of the above will do what you want?

These are all helpful tips and tricks, and I'll keep them
in mind. I really won't know if I can do the kind of things
I want with these until I try, I suppose. I don't have a
specific application in mind right now. Let me give you a
simple example of a CMS pipeline for illustrative purposes,
so that you will get some idea of what I'm talking about.

This is a REXX EXEC (i.e. a script written in the REXX
interpretive language) designed to run on the CMS operating
system running in a virtual machine under z/VM:

----------

'PIPE (ENDCHAR ?)', /* Declare pipeline endchar. */
' < USER DIRECT|', /* Read directory file. */
' NFIND *|', /* Discard comments. */
'LOCM: LOCATE /MDISK/|', /* Select MDISK records */
'FIN: FANINANY|', /* Collect MDISKs and LINKs. */
' > OUTPUT FILE A', /* Write them to a file. */
'?', /* End of stream. */
'LOCM: |', /* Non-MDISK cards to here */
' LOCATE /LINK/|', /* Select LINK cards. */
'FIN:' /* Send them to faninany. */
/* End of stream and pipe */

----------

This is all one command, as viewed by the REXX interpreter.
The stuff between /* and */ are comments. They are ignored
by the interpreter. Note that all lines end in a comma,
which is the continuation character, except the last. The
vertical bar, as with shell pipelines, is the stage separator.
The apostrophes serve as quote characters, just as in the shell.
PIPE is the name of the command to invoke. The rest of the
stuff is just arguments to the PIPE command. In other words,
the REXX interpreter itself has no concept of a pipeline.
It is just passing a string of characters to a CMS command
called PIPE. It is the PIPE command that understands what
a pipeline is. The question mark is defined as an
end-of-stream character. It marks the end of a section of
pipe, if you will.

The pipe consists of a number of stages. The stage numbers
and their names are as follows:

1. <
2. NFIND
3. LOCATE
4. FANINANY
5. >
6. LOCATE

Don't be confused by "<" and ">". They are not redirection
operators. There's no such thing as a redirection operator
in REXX or in CMS Pipelines. "<" and ">" are the names of
stages, just like the others. Stage 2 and stage 6 are two
different instances of the LOCATE stage. They have separate
inputs and outputs. LOCM and FIN are labels. They are
names given to some of the stages so that they can be
referenced later. You can tell they are labels because they
end with a colon.

Stage 1, "<", reads records (lines) from a file, the name
of which is USER DIRECT *. (The asterisk, meaning "on the
first accessed disk you can find it on" is implied by default.)
Input stream 1 is unconnected
and not used, since it is the first stage. Records from
the file are written to output stream 1.

Stage 2, NFIND, has its input stream 1 connected to output stream
1 of stage 1. Records which do not begin with an asterisk
in column 1 are written to output stream 1. Records which
do begin with an asterisk are written to output stream 2,
if it is connected. Otherwise, they are discarded.
Since output stream 2 is not connected, they are discarded.

Stage 3, LOCATE, has its input stream 1 connected to output
stream 1 of stage 2. Records containing the character string
"MDISK" are written to output stream 1. Records which do not
contain the character string "MDISK" are written to output
stream 2, if it is connected. Otherwise, they are discarded.
As we will later see, output stream 2 is connected; so these
records are not discarded. This stage is given the label
LOCM so that it can be referred to later.

Stage 4, FANINANY, has two connected input streams. Input
stream 1 is connected to output stream 1 from stage 3.
We will see later what is connected to input stream 2.
FANINANY reads from whichever input stream has a record
ready and writes it to output stream 1.

Stage 5, ">", has its input stream 1 connected to output
stream 1 of stage 4. ">" writes these records to a file,
whose name is OUTPUT FILE on the "A" disk. Output
stream 1 is unconnected, since it is followed by the
end-of-stream character. It is not used.

Stage 6, the second LOCATE stage, has its input stream 1
connected to output stream 2 of stage 3, since it is
preceded by "LOCM: |". Therefore input records from
stage 3 which do not contain the character string
"MDISK" end up coming in on input stream 1 of stage 6.
If any of these records contain the character string
"LINK", they are written to output stream 1. Records
which do not contain the character string "LINK" are
written to output stream 2, if it is connected. Otherwise,
they are discarded. Since output stream 2 is not
connected, these records are discarded. The trailing
string "|FIN:" indicates that ouput stream 1 is to be
connected to input stream 2 (since it is the second
reference to the label) of the FANINANY stage, stage 4.

This pipeline can be illustrated graphically as follows:

+---+ +-------+ +--------+ +----------+ +---+
|-| < |-->| NFIND |-->| LOCATE |------>| FANINANY |-->| > |-|
+---+ +-------+ +--------+ 1 1 +----------+ +---+
| 2 | 2
V A
| +--------+ |
LOCM:-->| LOCATE |-->-FIN:
+--------+

(Obviously, you must be viewing this using a non-proportional
font or it will look terrible.)

The net result is that, using a single pass through the
input file, records which contain the character string
MDISK or LINK, not including records that have been commented
out, are extracted from the input file and written to the
output file. This is a very simple example of a multi-stream
pipeline. Much more complex pipeline topologies can be
defined, as you can imagine. And there are lots of input
device stages, output device stages, filter stages, and
stages which do various other kinds of processing and
transformations.

Perhaps this gives you a flavor for CMS Pipelines. I don't
see anything like that available in shell pipelines.

Sorry for the long post, but I thought a specific example
would help.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

On Friday 05 February 2010 14:47:21 Stephen Powell wrote:
> On Fri, 5 Feb 2010 12:42:47 -0500 (EST), Bob McGowan wrote:
> Let me give you a
> simple example of a CMS pipeline for illustrative purposes,
> so that you will get some idea of what I'm talking about.
>

[snip: Long, but very useful description.]

> This pipeline can be illustrated graphically as follows:
>
> +---+ +-------+ +--------+ +----------+ +---+
> |-| < |-->| NFIND |-->| LOCATE |------>| FANINANY |-->| > |-|
> +---+ +-------+ +--------+ 1 1 +----------+ +---+
> | 2 | 2
> V A
> | +--------+ |
> LOCM:-->| LOCATE |-->-FIN:
> +--------+

mkfifo "LOCM" &&
mkfifo "FIN" &&
(
"<" | "NFIND" | "LOCATE" 2> "LOCM" | "FANINANY" 2< "FIN"| ">" &
"LOCATE" < "LOCM" > "FIN" &
wait
)

People with more shell magic than me might be able to simplify that to not use
fifos, but I figured they were fair game since you get stream labels. (That's
basically all a fifo is.)

Technically speaking, that's all one command, per the Single UNIX Specifation.
It matches the shell grammar start symbol "complete_command" documented
<http://www.opengroup.org/onlinepubs/007908775/xcu/chap2.html#tag_001_010_002>.

So, basically, we have different basic building blocks (I don't know a grep
variant that acts like LOCATE, for example), but the larger structure is very
similar.

Most UNIX/Linux utilities for the command line assume all input on fd 0, all
normal output on fd 1, and end-user notifications written to fd 2. However,
there's nothing actively preventing binaries from reading from fd 2, writing
to fd 0, or expecting i/o on fds 3 through 9. (IIRC, anything greater than 9
is reserved when using the shell.)
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/

- Bash question: get output as a variable?
 

Boyd Stephen Smith Jr. wrote:
> On Friday 05 February 2010 14:47:21 Stephen Powell wrote:
>> On Fri, 5 Feb 2010 12:42:47 -0500 (EST), Bob McGowan wrote:
>> Let me give you a
>> simple example of a CMS pipeline for illustrative purposes,
>> so that you will get some idea of what I'm talking about.
>>
>
> [snip: Long, but very useful description.]
>
>> This pipeline can be illustrated graphically as follows:
>>
>> +---+ +-------+ +--------+ +----------+ +---+
>> |-| < |-->| NFIND |-->| LOCATE |------>| FANINANY |-->| > |-|
>> +---+ +-------+ +--------+ 1 1 +----------+ +---+
>> | 2 | 2
>> V A
>> | +--------+ |
>> LOCM:-->| LOCATE |-->-FIN:
>> +--------+
>
> mkfifo "LOCM" &&
> mkfifo "FIN" &&
> (
> "<" | "NFIND" | "LOCATE" 2> "LOCM" | "FANINANY" 2< "FIN"| ">" &
> "LOCATE" < "LOCM" > "FIN" &
> wait
> )
>
> People with more shell magic than me might be able to simplify that to not use
> fifos, but I figured they were fair game since you get stream labels. (That's
> basically all a fifo is.)

I believe the use of fifo's in this context is required. Unless you're
willing to write the 'LOCATE' and 'FANINANY' tools first, to deal with
inputs on descriptors other than 0, 1 and 2.

Not something I'd care to do, personally. ;)

You might be able to do this using 'exec' to open and redirect the I/O
streams at the shell level, but the nature of the '|' in UNIX/Linux is
to read stdin and write stdout, so everything would need to ultimately
go through those two descriptors, which would disallow having the
"parallel" processing implied by the above diagram.

>
> Technically speaking, that's all one command, per the Single UNIX Specifation.
> It matches the shell grammar start symbol "complete_command" documented
> <http://www.opengroup.org/onlinepubs/007908775/xcu/chap2.html#tag_001_010_002>.
>
> So, basically, we have different basic building blocks (I don't know a grep
> variant that acts like LOCATE, for example), but the larger structure is very
> similar.
>
> Most UNIX/Linux utilities for the command line assume all input on fd 0, all
> normal output on fd 1, and end-user notifications written to fd 2. However,
> there's nothing actively preventing binaries from reading from fd 2, writing
> to fd 0, or expecting i/o on fds 3 through 9. (IIRC, anything greater than 9
> is reserved when using the shell.)

--
Bob McGowan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

On Friday 05 February 2010 16:33:12 Bob McGowan wrote:
> Boyd Stephen Smith Jr. wrote:
> > mkfifo "LOCM" &&
> > mkfifo "FIN" &&
> > (
> > "<" | "NFIND" | "LOCATE" 2> "LOCM" | "FANINANY" 2< "FIN"| ">" &
> > "LOCATE" < "LOCM" > "FIN" &
> > wait
> > )
>
> which would disallow having the
> "parallel" processing implied by the above diagram.

My example runs in parallel. It's virtually required when using fifos. They
aren't normal files -- only so much data can be written to their internal
buffer before further writes block.

In this case the shell actually starts at least 6 processes. One for each
"utility". Prior to doing the "exec" call in each process, it sets up the
pipes and redirection. It doesn't wait for the processes to finish (notice
the '&' at the end of the pipelines) until I use the wait shell-builtin.

When the "LOCATE" utility on the second line starts trying to read from stdin
(LOCM) it will block until the "LOCATE" utility on the second line writes data
to stderr (LOCM) and more-or-less immediately start working on that data as
soon as it arrives. Similarly, "FANINMANY" will block when trying to read
stderr (FIN) just until the "LOCATE" on the second line writes data out to
stdout (FIN).
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/

- Bash question: get output as a variable?
 

Boyd Stephen Smith Jr. wrote:
> On Friday 05 February 2010 16:33:12 Bob McGowan wrote:
>> Boyd Stephen Smith Jr. wrote:
>>> mkfifo "LOCM" &&
>>> mkfifo "FIN" &&
>>> (
>>> "<" | "NFIND" | "LOCATE" 2> "LOCM" | "FANINANY" 2< "FIN"| ">" &
>>> "LOCATE" < "LOCM" > "FIN" &
>>> wait
>>> )
>> which would disallow having the
>> "parallel" processing implied by the above diagram.
>
> My example runs in parallel. It's virtually required when using fifos. They
> aren't normal files -- only so much data can be written to their internal
> buffer before further writes block.
>
> In this case the shell actually starts at least 6 processes. One for each
> "utility". Prior to doing the "exec" call in each process, it sets up the
> pipes and redirection. It doesn't wait for the processes to finish (notice
> the '&' at the end of the pipelines) until I use the wait shell-builtin.
>
> When the "LOCATE" utility on the second line starts trying to read from stdin
> (LOCM) it will block until the "LOCATE" utility on the second line writes data
> to stderr (LOCM) and more-or-less immediately start working on that data as
> soon as it arrives. Similarly, "FANINMANY" will block when trying to read
> stderr (FIN) just until the "LOCATE" on the second line writes data out to
> stdout (FIN).

My point, exactly, though better said ;)

If someone wanted to, they could write a single program that would read
input/write output from/to two or more descriptors, and use it to create
a multi-path'ed I/O processing sequence, but writing it to prevent
blocking on one input/output, so it could continue working on the other
pair, would be a bit hairy.

The fifo node makes this sort of thing easy to do.

--
Bob McGowan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

- Bash question: get output as a variable?
 

On Fri, 5 Feb 2010 18:27:49 -0500 (EST), Bob McGowan wrote:
> If someone wanted to, they could write a single program that would read
> input/write output from/to two or more descriptors, and use it to create
> a multi-path'ed I/O processing sequence, but writing it to prevent
> blocking on one input/output, so it could continue working on the other
> pair, would be a bit hairy.
>
> The fifo node makes this sort of thing easy to do.

Well, as Mr. Spock would say,

Fascinating!

It appears that the combination of a shell pipeline and
fifos makes the implementation of a multi-streamed pipeline
possible. The problem is the lack of "stages",
especially the stream splitting and stream joining ones.

Thank you, Bob and Boyd, for your insights and expertise.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org