Hi,
Looking for a simple way to do a big copy at the command line. I
have a bunch of files (maybe 100 right now, but it will grow) that I
can find with locate and grep:
I need to copy these files to a new directory
(~mark/CorrelationTests) where I will modify what's in them before
running correlation tests on the contents.
How do I feed the output of the command above to cp at the command
line to get this done?
I've been playing with things like while & read but I can't get it right.
Thanks,
Mark
02-08-2011, 06:02 PM
Florian Philipp
copy a bunch of files...
Am 08.02.2011 19:27, schrieb Mark Knecht:
> Hi,
> Looking for a simple way to do a big copy at the command line. I
> have a bunch of files (maybe 100 right now, but it will grow) that I
> can find with locate and grep:
>
> c2stable ~ # locate Correlation | grep Builder | grep csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
> <SNIP>
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
> c2stable ~ #
>
> I need to copy these files to a new directory
> (~mark/CorrelationTests) where I will modify what's in them before
> running correlation tests on the contents.
>
> How do I feed the output of the command above to cp at the command
> line to get this done?
>
> I've been playing with things like while & read but I can't get it right.
>
> Thanks,
> Mark
>
-IARG tells xargs to replace the occurrence of ARG in the parameters
with the actual parameters read from stdin.
or
locate Correlation | grep Builder | grep csv | while read file; do
cp "$file" ~mark/CorrelationTests; done
BTW: Wouldn't grep 'Builder/.*.csv' match better (some intermediate
directory Builder, ending on .csv)?
Even easier:
locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests
Warning: I've not tested every line. Use with caution.
Hope this helps,
Florian Philipp
02-08-2011, 06:22 PM
Mark Knecht
copy a bunch of files...
On Tue, Feb 8, 2011 at 11:02 AM, Florian Philipp <lists@binarywings.net> wrote:
> Am 08.02.2011 19:27, schrieb Mark Knecht:
>> Hi,
>> * *Looking for a simple way to do a big copy at the command line. I
>> have a bunch of files (maybe 100 right now, but it will grow) that I
>> can find with locate and grep:
>>
>> c2stable ~ # locate Correlation | grep Builder | grep csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V1.csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V2.csv
>> /home/mark/Builder/TF/TF.D-17M-2009_06-2010_11/Correlation/TF.D-17M-2009_06-2010_11-V3.csv
>> <SNIP>
>> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V4.csv
>> /home/mark/Builder/TF/TF.D-31M-2009_06-2010_11/Correlation/TF.D-31M-2009_06-2010_11-V5.csv
>> c2stable ~ #
>>
>> * *I need to copy these files to a new directory
>> (~mark/CorrelationTests) where I will modify what's in them before
>> running correlation tests on the contents.
>>
>> * *How do I feed the output of the command above to cp at the command
>> line to get this done?
>>
>> * *I've been playing with things like while & read *but I can't get it right.
>>
>> Thanks,
>> Mark
>>
>
> locate Correlation | grep Builder | grep csv | xargs -IARG cp ARG
> ~mark/CorrelationTests
>
> -IARG tells xargs to replace the occurrence of ARG in the parameters
> with the actual parameters read from stdin.
>
Thanks! This worked nicely and is relatively easy to remember.
> or
>
> locate Correlation | grep Builder | grep csv | while read file; do
> cp "$file" ~mark/CorrelationTests; done
>
This is what I was trying to do but was unsuccessful.
> BTW: Wouldn't grep 'Builder/.*.csv' match better (some intermediate
> directory Builder, ending on .csv)?
>
Yes, that does seem to work. I guess that's grepping for the path of a
file starting with Builder and ending with CSV?
Good one.
> Even easier:
> locate ~mark/'*/Builder/*.csv' | xargs -IARG cp ARG ~mark/CorrelationTests
>
> Warning: I've not tested every line. Use with caution.
>
> Hope this helps,
> Florian Philipp
It did very much. Thanks!
Now to work on modifying the files, again with a loop for all the
files in ~mark/CorrelationTests
- Mark
02-09-2011, 01:51 AM
Harry Putnam
copy a bunch of files...
Mark Knecht <markknecht@gmail.com> writes:
>> locate Correlation | grep Builder | grep csv | while read file; do
>> cp "$file" ~mark/CorrelationTests; done
Just a minor point that would simplify the cmd by one cmd call.
You could use awk instead of 2 calls to grep. It might be a tiny bit
slower... but I doubt it would be noticeable at the proposed scale.
awk is basic tool on linux for cmd line work... its quite a bit more
powerful than grep, but of course more complicated.
(~mark/CorrelationTests) where I will modify what's in them before
running correlation tests on the contents.
* How do I feed the output of the command above to cp at the command
line to get this done?
* I've been playing with things like while & read *but I can't get it right.
Thanks,
Mark
Another way to do it is with find:
find /home/mark/Builder -type f -iname '*csv' -exec cp {} ~mark/CorrelationTests ;
If catching the *csv is not enough, you can use -ipath instead of -iname and do something like this:
-ipath '*Builder*csv'
this will match all the path.
Regards,
Kfir
02-09-2011, 10:56 AM
Neil Bothwick
copy a bunch of files...
On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
> Another way to do it is with find:
> find /home/mark/Builder -type f -iname '*csv' -exec cp {}
> ~mark/CorrelationTests ;
Replace ; with + for a faster process, as Mark said there are hundreds
of these files.
Or, if you use zsh instead of bash, it can be as simple as
cp Builder/**/*.csv CorrelationTests
--
Neil Bothwick
Learn from your parents' mistakes - use birth control!
02-09-2011, 11:09 AM
Helmut Jarausch
copy a bunch of files...
On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
> On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
>
> > Another way to do it is with find:
> > find /home/mark/Builder -type f -iname '*csv' -exec cp {}
> > ~mark/CorrelationTests ;
>
> Replace ; with + for a faster process, as Mark said there are
> hundreds
> of these files.
>
> Or, if you use zsh instead of bash, it can be as simple as
>
> cp Builder/**/*.csv CorrelationTests
>
There is a problem with this approach, though.
It can easily give "command line too long".
Helmut.
02-09-2011, 01:46 PM
Mark Knecht
copy a bunch of files...
On Wed, Feb 9, 2011 at 4:09 AM, Helmut Jarausch
<jarausch@igpm.rwth-aachen.de> wrote:
> On 02/09/2011 12:56:09 PM, Neil Bothwick wrote:
>> On Wed, 9 Feb 2011 13:38:37 +0200, Kfir Lavi wrote:
>>
>> > Another way to do it is with find:
>> > find /home/mark/Builder -type f -iname '*csv' -exec cp {}
>> > ~mark/CorrelationTests ;
>>
>> Replace ; with + for a faster process, as Mark said there are
>> hundreds
>> of these files.
>>
>> Or, if you use zsh instead of bash, it can be as simple as
>>
>> cp Builder/**/*.csv CorrelationTests
>>
> There is a problem with this approach, though.
> It can easily give "command line too long".
>
> Helmut.
Lots of interesting ideas. I use apps so often I've never become very
strong at the command line and yet people built this whole Linux
empire using it. It's very powerful.
One thing I didn't make clear in my original post - it didn't seem
important to confuse my real question which was the copy itself and
not locating the files - but which likely changes how well some of
these commands would work in my specific case was that the Builder
directory actually has _many_ CSV files ut specifically I needed only
the ones in the Correlation directories. Additionally, being that this
is stock & futures trading data, generally at a given time I need the
CSV files for a specific symbol, for instance in the original post:
I knew I wanted the Correlation directory but it turned out I had
other directories with that name on the system, so I added the grep
Builder to get me into the right tree and CSV to find only the CSV
files. However at that point I only had Russell futures data (TF.D) so
I didn't have to go further. Now, however, as I bring in Dow futures
(YM, YM.D) , S&P 500 futures (ES, ES.D), and NASDAQ futures (NQ, ND.D)
I just an an extra grep in and I'm there in terms of finding the files
I need for a certain test.
Additionally I have test results for other date ranges that will show
up soon, (2001-2005, or 2003-2011, etc. Additional greps are an easy
way to just winnow it down to a point where I'm finding what I need to
find.
This thread has given me a lot of new commands to go look at.
Thanks,
Mark
02-09-2011, 07:10 PM
Neil Bothwick
copy a bunch of files...
On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote:
> One thing I didn't make clear in my original post - it didn't seem
> important to confuse my real question which was the copy itself and
> not locating the files - but which likely changes how well some of
> these commands would work in my specific case was that the Builder
> directory actually has _many_ CSV files ut specifically I needed
> only the ones in the Correlation directories.
Bear in mind that locate only returns files in its database, not any
created since the last time its cron job was run. Find seems a more
appropriate tool for this task.
02-09-2011, 09:22 PM
Mark Knecht
copy a bunch of files...
On Wed, Feb 9, 2011 at 12:10 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 9 Feb 2011 06:46:34 -0800, Mark Knecht wrote:
>
>> One thing I didn't make clear in my original post - it didn't seem
>> important to confuse my real question which was the copy itself and
>> not locating the files - but which likely changes how well some of
>> these commands would work in my specific case was that the Builder
>> directory actually has _many_ CSV files ut specifically I needed
>> only the ones in the Correlation directories.
>
> Bear in mind that locate only returns files in its database, not any
> created since the last time its cron job was run. Find seems a more
> appropriate tool for this task.
>
Good point. As I bring these files down from a bunch of Windows VMs I
typically run updatedb before doing this, but find would probably be
safer.
copy a bunch of files...
