FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu User

 
 
LinkBack Thread Tools
 
Old 12-01-2009, 04:03 PM
Ray Parrish
 
Default Reading a variable line by line with while loop

Avi Greenbury wrote:
> Ray Parrish wrote:
>
>
>> Smoot Carl-Mitchell wrote:
>> Thank you! That worked great! Now I wonder is there is another great
>> sed command that will remove redundancy in the bash history, so no
>> command line is repeated in the final results in the $History
>> variable?
>>
>
> man uniq
>
> which requires you to sort it, too:
>
> man sort
>
> since it checks for repeated (consecutive) lines. If order matters,
> you'll need something more complicated.
>
> sed can probably be made to do it, but piping through sort and uniq is
> probably nicer.
>
>
> --
> Avi Greenbury
> http://aviswebsite.co.uk
> http://aviswebsite.co.uk/asking-questions
>
Thank you very much for the pointers! The following amended code is
working great!

History=""
# Sort the bash history into a temporary file so uniq can eliminate all
duplicate lines.
sort -d ~/.bash_history >"$DirectoryName/BashHistory.tmp"
# Read that file into a variable, and replace all space characters in
the file with __ [double underlines]
History=`sed 's/ /__/g' "$DirectoryName/BashHistory.tmp"`
# Write amended history back to temp file.
echo "$History" >"$DirectoryName/BashHistory.tmp"
# Eliminate redundant lines in the History variable.
History=`uniq -u "$DirectoryName/BashHistory.tmp"`

I didn't really want to write it back to file again, but I couldn't
figure out how to read from the $History variable with the uniq command,
I got the error that there was an extra parameter when i used the
following line for uniq.

HIstory=`unig -u $History`
uniq: extra operand `./AddURLtoSitemap.sh'

I would rather keep the data in the variable and work on it there,
instead of writing it back to file just to read it off the disk again.
That's a bit inefficient.

Later, Ray Parrish

--
The Future of Technology.
http://www.rayslinks.com/The%20Future%20of%20Technology.html
Ray's Links, a variety of links to usefull things, and articles by Ray.
http://www.rayslinks.com
Writings of "The" Schizophrenic, what it's like to be a schizo, and other
things, including my poetry.
http://www.writingsoftheschizophrenic.com



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 04:30 PM
Ray Parrish
 
Default Reading a variable line by line with while loop

Loc Greni wrote:
> 2009/12/1 Ray Parrish <crp@cmc.net>:
>
>> Ray Parrish wrote:
>>
>>> Loc Greni wrote:
>>>
>>>
>>>> 2009/12/1 Ray Parrish <crp@cmc.net>:
>>>>
>>>>
>>>>
>>>>> Hello,
>>>>>
>>>>> I have been using while loops to read files in line by line when I want
>>>>> to process things on a line oriented basis. Sometimes i have the set of
>>>>> lines I want to read through already in a variable, and have to write
>>>>> them to file before beginning to read line by line.
>>>>>
>>>>> I was wondering if it is possible to read a variable line by line
>>>>> somehow with a while loop? Would it possibly be faster than using a file
>>>>> to read from?
>>>>>
>>>>> I was thinking something along the lines of the following code might
>>>>> work, but have not tested it yet.
>>>>>
>>>>> while read ThisLine; do
>>>>> # process each line
>>>>> echo "$ThisLine"
>>>>> done < `echo "$Variable"`
>>>>>
>>>>> Is this something that might work, and would it be more efficient than
>>>>> writing to, and reading from a file? Is there some other way I've missed
>>>>> to make it work?
>>>>>
>>>>>
>>>>>
>>>> This will probably not work.
>>>>
>>>> echo "$Variable" | while read
>>>> ...
>>>> done
>>>>
>>>> has better chances of working.
>>>>
>>>> Loc
>>>>
>>>>
>>>>
>>> Well, that one test worked, but I cannot get the following to work -
>>>
>>> # Read bash history into a variable for use with combobox call
>>> History=""
>>> BashHistory=`cat ~/.bash_history`
>>> echo "$BashHistory" | while read ThisCommand; do
>>> # Replace spaces in each command line with __ [double underlines]
>>> ThisCommand=${ThisCommand// /__}
>>> # Concatenate the results to the Command variable
>>> History="$History $ThisCommand"
>>> done
>>> echo "History - $History"
>>>
>>> The last echo command returns nothing, but if I put an echo command in
>>> the loop either before, or after the replace spaces command, it echoes
>>> every line of the variable to the console.
>>>
>>> Does anyone have any idea why it's not working because I am stumped...
>>>
>>> Later, Ray Parrish
>>>
>>>
>> Hello again,
>>
>> OK, I understand that the | is causing a sub shell to execute for the
>> echo command, so the variable $History even 'though declared globally,
>> is local to the loop with the pipe, and therefore not available after
>> the loop exits.
>>
>> I have tested this by putting an echo "History - $History" command right
>> after the concatenation line in the loop. This causes some pretty slow
>> execution as the extremely long string get echoed over and over again,
>> but showed that the variable was indeed getting updated each time
>> through the loop.
>>
>> Does anyone know how to fix my code so it can access the value of the
>> $History variable after the loop exits?
>>
>
> You might need to adjust the ; but something along the
> lines of:
>
> echo "$BashHistory" | (while read ThisCommand; do
> ThisCommand=${ThisCommand// /__};
> History="$History $ThisCommand";
> done;
> echo "History - $History")
>
This worked great! Thanks for teaching me how to read a variable line by
line, the semi-colons did the trick, and now I can access the variable
contents after the loop is over.
> I don't know what you want to do but:
>
> a=`cat file`
> echo "$a" | whatever
>
> is a strange way to do
>
Well, I often find myself with a file already loaded in a variable, or
have built up a line oriented set of data in a variable without it
having been in a file in the first place, so I am under the half-baked
impression that handling the variable's data directly instead of
flushing to, and re-reading from a file would be more efficient.

Thanks again for the help.

Later, Ray Parrish

--
The Future of Technology.
http://www.rayslinks.com/The%20Future%20of%20Technology.html
Ray's Links, a variety of links to usefull things, and articles by Ray.
http://www.rayslinks.com
Writings of "The" Schizophrenic, what it's like to be a schizo, and other
things, including my poetry.
http://www.writingsoftheschizophrenic.com



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 04:42 PM
Ray Parrish
 
Default Reading a variable line by line with while loop

Ray Parrish wrote:
> Avi Greenbury wrote:
>
>> Ray Parrish wrote:
>>
>>
>>
>>> Smoot Carl-Mitchell wrote:
>>> Thank you! That worked great! Now I wonder is there is another great
>>> sed command that will remove redundancy in the bash history, so no
>>> command line is repeated in the final results in the $History
>>> variable?
>>>
>>>
>> man uniq
>>
>> which requires you to sort it, too:
>>
>> man sort
>>
>> since it checks for repeated (consecutive) lines. If order matters,
>> you'll need something more complicated.
>>
>> sed can probably be made to do it, but piping through sort and uniq is
>> probably nicer.
>>
>>
>> --
>> Avi Greenbury
>> http://aviswebsite.co.uk
>> http://aviswebsite.co.uk/asking-questions
>>
>>
> Thank you very much for the pointers! The following amended code is
> working great!
>
> History=""
> # Sort the bash history into a temporary file so uniq can eliminate all
> duplicate lines.
> sort -d ~/.bash_history >"$DirectoryName/BashHistory.tmp"
> # Read that file into a variable, and replace all space characters in
> the file with __ [double underlines]
> History=`sed 's/ /__/g' "$DirectoryName/BashHistory.tmp"`
> # Write amended history back to temp file.
> echo "$History" >"$DirectoryName/BashHistory.tmp"
> # Eliminate redundant lines in the History variable.
> History=`uniq -u "$DirectoryName/BashHistory.tmp"`
>
> I didn't really want to write it back to file again, but I couldn't
> figure out how to read from the $History variable with the uniq command,
> I got the error that there was an extra parameter when i used the
> following line for uniq.
>
> HIstory=`unig -u $History`
> uniq: extra operand `./AddURLtoSitemap.sh'
>
> I would rather keep the data in the variable and work on it there,
> instead of writing it back to file just to read it off the disk again.
> That's a bit inefficient.
>
> Later, Ray Parrish
>
Hello again,

I like the fact that it only took three lines of code to sort that out
and reduce the redundancy, but I have decided the order of the original
bash history is important, and am working on a way to test each line as
read in the loop to see if it already exists in the variable.

I need to complete the following if statement somehow.

BashHistory=`cat ~/.bash_history`
echo "$BashHistory" | (while read ThisCommand; do
ThisCommand=${ThisCommand// /__};
# If this input string does not already exist in $History
if [ ! $ThisCommand in $History ]
then
History="$History $ThisCommand";
fi
done;
echo "History - $History")

Anyway, that's what I'm working on now, so thanks again for all the help
you people have been being for this novice bash scripter. 8-)

Later, Ray Parrish


--
The Future of Technology.
http://www.rayslinks.com/The%20Future%20of%20Technology.html
Ray's Links, a variety of links to usefull things, and articles by Ray.
http://www.rayslinks.com
Writings of "The" Schizophrenic, what it's like to be a schizo, and other
things, including my poetry.
http://www.writingsoftheschizophrenic.com



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 08:51 PM
Alan McKay
 
Default Reading a variable line by line with while loop

> echo "$Variable" | while read
> ...
> done
>
> * *has better chances of working.

This method has gotchas though, and I try to avoid it. The pipe
causes the while to execute in a sub shell and so any variable changes
you make inside the loop will be lost when you get out of the loop.
For this reason I often actually create a file and do the redirect as
per the original post

e.g.

while stuff
do
done < file




--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 09:01 PM
PleegWat
 
Default Reading a variable line by line with while loop

Alan McKay wrote:
>> echo "$Variable" | while read
>> ...
>> done
>>
>> has better chances of working.
>
> This method has gotchas though, and I try to avoid it. The pipe
> causes the while to execute in a sub shell and so any variable changes
> you make inside the loop will be lost when you get out of the loop.
> For this reason I often actually create a file and do the redirect as
> per the original post
>
> e.g.
>
> while stuff
> do
> done < file

Another (bash-specific) solution:

while condition
do stuff
done < <(command)

The <(command) construct runs `command`, redirects stdout to a pipe, and
returns the name of the named pipe.

This construct has the disadvantages of being bash-specific and
resulting in hard-to-read code.

Two more solutions specifically for input from a variable: Heredoc

while condition
do stuff
done <<EOF
$variable
EOF

Or herestring (bash-specific)

while condition
do stuff
done <<< "$variable"

PleegWat

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 09:34 PM
James Michael Fultz
 
Default Reading a variable line by line with while loop

* Ray Parrish <crp@cmc.net> [2009-12-01 09:42 -0800]:
> I like the fact that it only took three lines of code to sort that out
> and reduce the redundancy, but I have decided the order of the original
> bash history is important, and am working on a way to test each line as
> read in the loop to see if it already exists in the variable.
>
> I need to complete the following if statement somehow.
>
> BashHistory=`cat ~/.bash_history`
> echo "$BashHistory" | (while read ThisCommand; do
> ThisCommand=${ThisCommand// /__};
> # If this input string does not already exist in $History
> if [ ! $ThisCommand in $History ]
> then
> History="$History $ThisCommand";
> fi
> done;
> echo "History - $History")

If you don't need to reuse your variables later, this should run faster.

awk '
END {
print "History - "
for (k in a) {
gsub(/ /, "__", k); print k
}
}
{ a[$0] }'
~/.bash_history

> Anyway, that's what I'm working on now, so thanks again for all the help
> you people have been being for this novice bash scripter. 8-)
>
> Later, Ray Parrish

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 10:29 PM
Ray Parrish
 
Default Reading a variable line by line with while loop

PleegWat wrote:
> Alan McKay wrote:
>
>>> echo "$Variable" | while read
>>> ...
>>> done
>>>
>>> has better chances of working.
>>>
>> This method has gotchas though, and I try to avoid it. The pipe
>> causes the while to execute in a sub shell and so any variable changes
>> you make inside the loop will be lost when you get out of the loop.
>> For this reason I often actually create a file and do the redirect as
>> per the original post
>>
>> e.g.
>>
>> while stuff
>> do
>> done < file
>>
>
> Another (bash-specific) solution:
>
> while condition
> do stuff
> done < <(command)
>
> The <(command) construct runs `command`, redirects stdout to a pipe, and
> returns the name of the named pipe.
>
> This construct has the disadvantages of being bash-specific and
> resulting in hard-to-read code.
>
> Two more solutions specifically for input from a variable: Heredoc
>
> while condition
> do stuff
> done <<EOF
> $variable
> EOF
>
> Or herestring (bash-specific)
>
> while condition
> do stuff
> done <<< "$variable"
>
> PleegWat
>
Hello again,

Well, I'm making some progress. I have the following code which works to
a point.

while read ThisCommand; do
ThisCommand=${ThisCommand// /__};
if `echo "${History}" | grep "${ThisCommand}" 1>/dev/null 2>&1`
then
echo "nothing" >/dev/null
else
History="$History $ThisCommand";
fi
done < <(cat ~/.bash_history)
Command=`Xdialog --stdout --title "Ray's Bash Manager - Select
Command to Copy" --cancel-label "Exit" --combobox "Select a Command to
Copy to the Clipboard." 0 60 $History`

The if statement is meant to catch duplicate lines, and it does, unless
they contain quotes in them, and then they get duplicated in the output.
Is there some way to trap even the lines with quotes in them?

Thanks again, Ray Parrish
--

The Future of Technology.
http://www.rayslinks.com/The%20Future%20of%20Technology.html
Ray's Links, a variety of links to usefull things, and articles by Ray.
http://www.rayslinks.com
Writings of "The" Schizophrenic, what it's like to be a schizo, and other
things, including my poetry.
http://www.writingsoftheschizophrenic.com



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 11:33 PM
NoOp
 
Default Reading a variable line by line with while loop

On 12/01/2009 01:40 AM, Ray Parrish wrote:
> Hello,
>
> I have been using while loops to read files in line by line when I want
> to process things on a line oriented basis. Sometimes i have the set of
> lines I want to read through already in a variable, and have to write
> them to file before beginning to read line by line.
>
> I was wondering if it is possible to read a variable line by line
> somehow with a while loop? Would it possibly be faster than using a file
> to read from?
>
> I was thinking something along the lines of the following code might
> work, but have not tested it yet.
>
> while read ThisLine; do
> # process each line
> echo "$ThisLine"
> done < `echo "$Variable"`
>
> Is this something that might work, and would it be more efficient than
> writing to, and reading from a file? Is there some other way I've missed
> to make it work?
>
> Thanks for any help you can be.
>
> Later, Ray Parrish
>

I think that your continued learning experience with bash/command
line/scripts et al is educational (to some), but what/how are all of
these posts from you related to Ubuntu technical support, and more
specifically to this list?



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 11:38 PM
Alan McKay
 
Default Reading a variable line by line with while loop

On Tue, Dec 1, 2009 at 7:33 PM, NoOp <glgxg@sbcglobal.net> wrote:

> I think that your continued learning experience with bash/command
> line/scripts et al is educational (to some), but what/how are all of
> these posts from you related to Ubuntu technical support, and more
> specifically to this list?

It runs on Ubuntu doesn't it?

If Ubuntu wants to claim to be "enterprise grade" then it must step up
to the plate and offer that sort of support. Yes, I know this is
free. But politeness and common points of manners go a long way.

And incidentally, the CentOS list answers these sorts of questions
with joy and dignity. Perhaps the OP needs to switch OSes.

cheers,
-Alan



--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 12-01-2009, 11:45 PM
Ray Parrish
 
Default Reading a variable line by line with while loop

Ray Parrish wrote:
> PleegWat wrote:
>
>> Alan McKay wrote:
>>
>>
>>>> echo "$Variable" | while read
>>>> ...
>>>> done
>>>>
>>>> has better chances of working.
>>>>
>>>>
>>> This method has gotchas though, and I try to avoid it. The pipe
>>> causes the while to execute in a sub shell and so any variable changes
>>> you make inside the loop will be lost when you get out of the loop.
>>> For this reason I often actually create a file and do the redirect as
>>> per the original post
>>>
>>> e.g.
>>>
>>> while stuff
>>> do
>>> done < file
>>>
>>>
>> Another (bash-specific) solution:
>>
>> while condition
>> do stuff
>> done < <(command)
>>
>> The <(command) construct runs `command`, redirects stdout to a pipe, and
>> returns the name of the named pipe.
>>
>> This construct has the disadvantages of being bash-specific and
>> resulting in hard-to-read code.
>>
>> Two more solutions specifically for input from a variable: Heredoc
>>
>> while condition
>> do stuff
>> done <<EOF
>> $variable
>> EOF
>>
>> Or herestring (bash-specific)
>>
>> while condition
>> do stuff
>> done <<< "$variable"
>>
>> PleegWat
>>
>>
> Hello again,
>
> Well, I'm making some progress. I have the following code which works to
> a point.
>
> while read ThisCommand; do
> ThisCommand=${ThisCommand// /__};
> if `echo "${History}" | grep "${ThisCommand}" 1>/dev/null 2>&1`
> then
> echo "nothing" >/dev/null
> else
> History="$History $ThisCommand";
> fi
> done < <(cat ~/.bash_history)
> Command=`Xdialog --stdout --title "Ray's Bash Manager - Select
> Command to Copy" --cancel-label "Exit" --combobox "Select a Command to
> Copy to the Clipboard." 0 60 $History`
>
> The if statement is meant to catch duplicate lines, and it does, unless
> they contain quotes in them, and then they get duplicated in the output.
> Is there some way to trap even the lines with quotes in them?
>
> Thanks again, Ray Parrish
>
Hello again,

I have found an if statement that reduces the duplicates to only two
pairs for the entire file. That's pretty good considering the number of
actual duplicates there were in the file to begin with. Here is the code
for that one -

function CompressHistory {
BashHistory=`cat ~/.bash_history`
while read ThisCommand; do
ThisCommand=${ThisCommand// /__};
if [[ "$History" =~ "$ThisCommand" ]]
then
echo "nothing" >/dev/null
else
History="$History $ThisCommand";
fi
done <<< "$BashHistory"
echo "$History"
}

This next bit of code has me frazzled trying to figure out the proper
regular expression to say "if this string exists within the larger
string". Nothing I have tried so far has worked, so here is the code,
and maybe someone can correct my regular expression so it works to weed
out duplicates as well.

BashHistory=`cat ~/.bash_history`
while read ThisCommand; do
ThisCommand=${ThisCommand// /__};
if [[ "$History" == [.]*$ThisCommand[.]* ]]
then
echo "nothing" >/dev/null
else
History="$History $ThisCommand";
fi
done <<< "$BashHistory"

From what I have read the dot is supposed to match any character, and
then the * specifies that any number of characters can appear in that
position. I'm just realizing that would specify perhaps one character
only, repeated to the ends of the large string before it would match.

I do not know how to specify that the whole front, and the whole back of
the variable should be the same with the matching string in the middle
of them.

Thanks again for any help you can be. Later, Ray Parrish

--
The Future of Technology.
http://www.rayslinks.com/The%20Future%20of%20Technology.html
Ray's Links, a variety of links to usefull things, and articles by Ray.
http://www.rayslinks.com
Writings of "The" Schizophrenic, what it's like to be a schizo, and other
things, including my poetry.
http://www.writingsoftheschizophrenic.com



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 

Thread Tools




All times are GMT. The time now is 10:09 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org