tar - ssh - standard out
 

Hi

I have a process that creates 'some data' and outputs this to standard
out and i want to shift this data over ssh to a remote box without
ever writing anything locally. I have been experimenting with tar to
create the archive as the i dont know what the contents of 'some data'
might be so i just need to capture it and output it on the other side.

I have been trying with

$ tar czf - . | ssh -q 192.168.122.2 "tar xzf -"

and this works fine to create an archive of this '.' directory and
pipe that over to the other side but i want to take standard out
so....

$ tar czf - `the thing that generates standard out here` | ssh -q
192.168.122.2 "tar xzf -"

Would that work or is there a better way to get this over to the other
side? It needs to be a data stream though so things like scp and rsync
are no good and i need to know what the command is on the remote side
being run so that i can restrict this in the ssh public key on the
remote side.

thanks
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

Tom Brown wrote:
> Hi
>
> I have a process that creates 'some data' and outputs this to standard
> out and i want to shift this data over ssh to a remote box without
> ever writing anything locally. I have been experimenting with tar to
> create the archive as the i dont know what the contents of 'some data'
> might be so i just need to capture it and output it on the other side.
>
> I have been trying with
>
> $ tar czf - . | ssh -q 192.168.122.2 "tar xzf -"
>
> and this works fine to create an archive of this '.' directory and
> pipe that over to the other side but i want to take standard out
> so....
>
> $ tar czf - `the thing that generates standard out here` | ssh -q
> 192.168.122.2 "tar xzf -"
>
> Would that work or is there a better way to get this over to the other
> side? It needs to be a data stream though so things like scp and rsync
> are no good and i need to know what the command is on the remote side
> being run so that i can restrict this in the ssh public key on the
> remote side.
>

Why do you need any other process involved to work with a data stream? If you
want to collect it to a remote file, you can | ssh remotehost 'cat >
path_to_file'. Just be sure to quote the redirection so it happens on the
remote side.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

On 08/12/2010 05:33 AM, Les Mikesell wrote:
> Why do you need any other process involved to work with a data stream? If you
> want to collect it to a remote file, you can | ssh remotehost 'cat>
> path_to_file'. Just be sure to quote the redirection so it happens on the
> remote side.
>
>

At a guess it's the compression he is after. Over a slow link it could
make a substantial difference.

--
Benjamin Franz
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

At Thu, 12 Aug 2010 13:11:21 +0100 CentOS mailing list <centos@centos.org> wrote:

>
> Hi
>
> I have a process that creates 'some data' and outputs this to standard
> out and i want to shift this data over ssh to a remote box without
> ever writing anything locally. I have been experimenting with tar to
> create the archive as the i dont know what the contents of 'some data'
> might be so i just need to capture it and output it on the other side.
>
> I have been trying with
>
> $ tar czf - . | ssh -q 192.168.122.2 "tar xzf -"
>
> and this works fine to create an archive of this '.' directory and
> pipe that over to the other side but i want to take standard out
> so....
>
> $ tar czf - `the thing that generates standard out here` | ssh -q
> 192.168.122.2 "tar xzf -"
>
> Would that work or is there a better way to get this over to the other
> side? It needs to be a data stream though so things like scp and rsync
> are no good and i need to know what the command is on the remote side
> being run so that i can restrict this in the ssh public key on the
> remote side.

Why not just do

`the thing that generates standard out here` | ssh -q 192.168.122.2 dd of=somethin


eg

find . | ssh -q 192.168.122.2 dd of=find.out

You don't need tar for anything.

>
> thanks
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

--
Robert Heller -- Get the Deepwoods Software FireFox Toolbar!
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

At Thu, 12 Aug 2010 06:05:25 -0700 CentOS mailing list <centos@centos.org> wrote:

>
> On 08/12/2010 05:33 AM, Les Mikesell wrote:
> > Why do you need any other process involved to work with a data stream? If you
> > want to collect it to a remote file, you can | ssh remotehost 'cat>
> > path_to_file'. Just be sure to quote the redirection so it happens on the
> > remote side.
> >
> >
>
> At a guess it's the compression he is after. Over a slow link it could
> make a substantial difference.

Just add gzip (or bzip2) to the pipeline:

program | bzip2 | ssh -q remote-host 'bunzip2 | remote-program'

>

--
Robert Heller -- Get the Deepwoods Software FireFox Toolbar!
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

Robert Heller wrote, On 08/12/2010 09:18 AM:
> At Thu, 12 Aug 2010 06:05:25 -0700 CentOS mailing list <centos@centos.org> wrote:
>
>> On 08/12/2010 05:33 AM, Les Mikesell wrote:
>>> Why do you need any other process involved to work with a data stream? If you
>>> want to collect it to a remote file, you can | ssh remotehost 'cat>
>>> path_to_file'. Just be sure to quote the redirection so it happens on the
>>> remote side.
>>>
>>>
>> At a guess it's the compression he is after. Over a slow link it could
>> make a substantial difference.
>
> Just add gzip (or bzip2) to the pipeline:
>
> program | bzip2 | ssh -q remote-host 'bunzip2 | remote-program'
>
>

or even easier (though maybe not as good a compression as bzip would get if dealing with text only)
program | ssh -C -q remote-host 'remote-program'

--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

On Thu, Aug 12, 2010 at 09:18:31AM -0400, Robert Heller wrote:

> program | bzip2 | ssh -q remote-host 'bunzip2 | remote-program'

If you're gonna put a compression tool in the pipeline then I recommend
you ensure ssh's own on-the-wire compression is turned off 'cos otherwise
you're potentially wasting CPU cycles.
ssh -q -o 'Compression no' remote-host

Yes; this may be the default value but it's always a good thing to
ensure sane values are used in cases like this :-)

--

rgds
Stephen
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

> Why not just do
>
> `the thing that generates standard out here` | ssh -q 192.168.122.2 dd of=somethin
>
>
> eg
>
> find . | ssh -q 192.168.122.2 dd of=find.out
>
> You don't need tar for anything.


alas the thing that generates the output creates 5 or 6 seperate
streams in sequence that generate 5 or 6 log files but i dont know in
advance the names of these logs.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

On 8/12/2010 8:46 AM, Tom Brown wrote:
>> Why not just do
>>
>> `the thing that generates standard out here` | ssh -q 192.168.122.2 dd of=somethin
>>
>>
>> eg
>>
>> find . | ssh -q 192.168.122.2 dd of=find.out
>>
>> You don't need tar for anything.
>
>
> alas the thing that generates the output creates 5 or 6 seperate
> streams in sequence that generate 5 or 6 log files but i dont know in
> advance the names of these logs.

You'll have to explain how the streams get sorted out locally before
anyone can help you do it remotely. Maybe the program itself could
piped each stream through a separate ssh instance. Or if you can wait
for output to complete, collect all the files in an otherwise empty
directory and rsync the whole thing to the remote.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

tar - ssh - standard out
 

At Thu, 12 Aug 2010 14:46:49 +0100 CentOS mailing list <centos@centos.org> wrote:

>
> > Why not just do
> >
> > `the thing that generates standard out here` | ssh -q 192.168.122.2 dd of=somethin
> >
> >
> > eg
> >
> > find . | ssh -q 192.168.122.2 dd of=find.out
> >
> > You don't need tar for anything.
>
>
> alas the thing that generates the output creates 5 or 6 seperate
> streams in sequence that generate 5 or 6 log files but i dont know in
> advance the names of these logs.

So the thing (program) does not write to stdout itself? It it does '5
or 6' fopen("<random>.log","w")s? Well, then you need to do:

(mkdir temp && cd temp && thing && tar czvf - . |
ssh -q 192.168.122.2 tar xvf -) && rm -rf temp

And yes, the log files will be written to the local disk before being
transfered. There is not really anyway around this, unless you were
will / able to rewrite 'thing' to work differently.

> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

--
Robert Heller -- 978-544-6933
Deepwoods Software -- Download the Model Railroad System
http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows
heller@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos