Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian User (http://www.linux-archive.org/debian-user/)
-   -   scripting question (http://www.linux-archive.org/debian-user/30290-scripting-question.html)

Matus UHLAR - fantomas 01-06-2008 04:38 PM

scripting question
 
On 21.09.07 08:18, Michael Martinell wrote:
> My script is as follows:
> #!/bin/sh
>
> TERM=vt100
> export TERM

forcing TERM in script is very bad idea, and in this script also useless.

> date && echo " Spam Count" && /bin/more /var/log/syslog | /bin/grep -c
> 'identified spam' && echo " " && echo "Clean Message Count " && /bin/more
> /var/log/syslog | /bin/grep -c 'clean message'

date

echo -n "Spam Count "
grep -Fc 'identified spam' /var/log/syslog

echo -n "Ham Count "
grep -Fc 'clean message' /var/log/syslog

... easier to read

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"They say when you play that M$ CD backward you can hear satanic messages."
"That's nothing. If you play it forward it will install Windows."


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Patton Echols 02-14-2009 01:18 AM

Scripting Question
 
I have a fairly massive flat file, comma delimited, that I want to
extract info from. Specifically, I want to extract the first and last
name and email addresses for those who have them to a new file with just
that info. (The windows database program that this comes from simply
will not do it) I can grep the file for the @ symbol to at least
exclude the lines without an email address (or the @ symbol in the notes
field) But if I can figure this out, I can also adapt what I learn for
the next time. Can anyone point me in the right direction for my "light
reading?"

By the way, I used 'head' to get the first line, with the field names.
This is the first of about 2300 records, the reason not to do it by hand.

patton@laptop:~$ head -1 contacts.txt
"Business Title","First Name","Middle Name","Last Name","","Business
Company Name","","Business Title","Business Street 1","Business Street
2","Business Street 3","Business City","Business State","Business
Zip","Business Country","Home Street 1","Home Street 2","Home Street
3","Home City","Home State","Home Zip","Home Country","Other Street
1","Other Street 2","Other Street 3","Other City","Other State","Other
Zip","Other Country","Assistant Phone","Business Fax Number","Business
Phone","Business 2 Phone","","Car Phone","","Home Fax Number","Home
Phone","Home 2 Phone","ISDN Phone","Mobile Phone","Other Fax
Number","Other Phone","Pager
Phone","","","","","","","","","","","","","Busine ss Email","","Home
Email","","Other
Email","","","","","","","","","","","","Notes","" ,"","","","","","","","","","","","","Business
Web Page"


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Chris Mohler 02-14-2009 01:41 AM

Scripting Question
 
On Sat, Feb 14, 2009 at 8:18 PM, Patton Echols <p.echols@comcast.net> wrote:
> I have a fairly massive flat file, comma delimited, that I want to
> extract info from. Specifically, I want to extract the first and last
> name and email addresses for those who have them to a new file with just
> that info. (The windows database program that this comes from simply
> will not do it) I can grep the file for the @ symbol to at least
> exclude the lines without an email address (or the @ symbol in the notes
> field) But if I can figure this out, I can also adapt what I learn for
> the next time. Can anyone point me in the right direction for my "light
> reading?"

Maybe this will help (a good start anyway):
#===========================
#!/usr/bin/env python

import csv

# Open CSV of ZIP code data
file = open("your filename here", 'r')
csv = csv.reader(file)

for row in csv:
do something....
#=======================

if you replace "do something" with "print row[0[", it will print the
first column, "print row[1]" the second column - you get the idea ;)

If you get an error about csv - check that the python-csv package is
installed...

Chris

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Lucio M Nicolosi 02-14-2009 01:44 AM

Scripting Question
 
Patton Echols wrote:
> I have a fairly massive flat file, comma delimited, that I want to
> extract info from. Specifically, I want to extract the first and last
> name and email addresses for those who have them to a new file with just
> that info. (The windows database program that this comes from simply
> will not do it) I can grep the file for the @ symbol to at least
> exclude the lines without an email address (or the @ symbol in the notes
> field) But if I can figure this out, I can also adapt what I learn for
> the next time. Can anyone point me in the right direction for my "light
> reading?"
>
> By the way, I used 'head' to get the first line, with the field names.
> This is the first of about 2300 records, the reason not to do it by hand.
>
> patton@laptop:~$ head -1 contacts.txt
> "Business Title","First Name","Middle Name","Last Name","","Business
> Company Name","","Business Title","Business Street 1","Business Street
> 2","Business Street 3","Business City","Business State","Business
> Zip","Business Country","Home Street 1","Home Street 2","Home Street
> 3","Home City","Home State","Home Zip","Home Country","Other Street
> 1","Other Street 2","Other Street 3","Other City","Other State","Other
> Zip","Other Country","Assistant Phone","Business Fax Number","Business
> Phone","Business 2 Phone","","Car Phone","","Home Fax Number","Home
> Phone","Home 2 Phone","ISDN Phone","Mobile Phone","Other Fax
> Number","Other Phone","Pager
> Phone","","","","","","","","","","","","","Busine ss Email","","Home
> Email","","Other
> Email","","","","","","","","","","","","Notes","" ,"","","","","","","","","","","","","Business
> Web Page"
>
>
>
Sure it's not an elegant solution, but couldn't you just import the
comma delimited data to a worksheet and edit it? OpenOffice or Symphony
would do it.

L.

--
Lucio M. Nicolosi, Eng.
São Paulo - Brazil
Lat.: 23°34'4.79"S
Long.: 46°39'59.53"W

email: lmnicolosi@gmail.com
skype: lmnicolosi1

Linux Registered User #481505
http://counter.li.org/




--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Corey Bettenhausen 02-14-2009 02:07 AM

Scripting Question
 
Patton Echols wrote:
> I have a fairly massive flat file, comma delimited, that I want to
> extract info from. Specifically, I want to extract the first and last
[snip]
Check out awk, particularly the -F switch to pluck out the desired
fields. The grep for the "@" to make sure the email addresses exist.
Something like:
awk -F"," '{print $1,$4,$6}' file.txt | grep "@" > out_file.txt
-Corey

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Patton Echols 02-14-2009 02:49 AM

Scripting Question
 
On 02/13/2009 06:44 PM, Lucio M Nicolosi wrote:
> Patton Echols wrote:
>
>> I have a fairly massive flat file, comma delimited, that I want to
>> extract info from. Specifically, I want to extract the first and last
>> name and email addresses for those who have them to a new file with just
>> that info. (The windows database program that this comes from simply
>> will not do it) I can grep the file for the @ symbol to at least
>> exclude the lines without an email address (or the @ symbol in the notes
>> field) But if I can figure this out, I can also adapt what I learn for
>> the next time. Can anyone point me in the right direction for my "light
>> reading?"
>>
>> By the way, I used 'head' to get the first line, with the field names.
>> This is the first of about 2300 records, the reason not to do it by hand.
>>
>> patton@laptop:~$ head -1 contacts.txt
>> "Business Title","First Name","Middle Name","Last Name","","Business
>> Company Name","","Business Title","Business Street 1","Business Street
>> 2","Business Street 3","Business City","Business State","Business
>> Zip","Business Country","Home Street 1","Home Street 2","Home Street
>> 3","Home City","Home State","Home Zip","Home Country","Other Street
>> 1","Other Street 2","Other Street 3","Other City","Other State","Other
>> Zip","Other Country","Assistant Phone","Business Fax Number","Business
>> Phone","Business 2 Phone","","Car Phone","","Home Fax Number","Home
>> Phone","Home 2 Phone","ISDN Phone","Mobile Phone","Other Fax
>> Number","Other Phone","Pager
>> Phone","","","","","","","","","","","","","Busine ss Email","","Home
>> Email","","Other
>> Email","","","","","","","","","","","","Notes","" ,"","","","","","","","","","","","","Business
>> Web Page"
>>
>>
>>
>>
> Sure it's not an elegant solution, but couldn't you just import the
> comma delimited data to a worksheet and edit it? OpenOffice or Symphony
> would do it.
>
> L.
>
>
I could, but I end up having to do something like this periodically. I
hoped to learn a quicker way to approach it. Since I am on a deadline
this time around, I may grep the file as I said, then use a OO-calc to
delete the columns I don't want.


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Patton Echols 02-14-2009 02:52 AM

Scripting Question
 
On 02/13/2009 06:41 PM, Chris Mohler wrote:
> On Sat, Feb 14, 2009 at 8:18 PM, Patton Echols <p.echols@comcast.net> wrote:
>
>> I have a fairly massive flat file, comma delimited, that I want to
>> extract info from. Specifically, I want to extract the first and last
>> name and email addresses for those who have them to a new file with just
>> that info. (The windows database program that this comes from simply
>> will not do it) I can grep the file for the @ symbol to at least
>> exclude the lines without an email address (or the @ symbol in the notes
>> field) But if I can figure this out, I can also adapt what I learn for
>> the next time. Can anyone point me in the right direction for my "light
>> reading?"
>>
>
> Maybe this will help (a good start anyway):
> #===========================
> #!/usr/bin/env python
>
> import csv
>
> # Open CSV of ZIP code data
> file = open("your filename here", 'r')
> csv = csv.reader(file)
>
> for row in csv:
> do something....
> #=======================
>
> if you replace "do something" with "print row[0[", it will print the
> first column, "print row[1]" the second column - you get the idea ;)
>
> If you get an error about csv - check that the python-csv package is
> installed...
>
> Chris
>
>
Is there a place where I can find the syntax for such a thing?

I like the idea of having "do something" be: pass one, print column 3,
column 4, column 12, column 13 and then on pass two, print the rows
where col 3 and 4 of the result have email addresses.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

Lucio M Nicolosi 02-14-2009 03:06 AM

Scripting Question
 
Patton Echols wrote:
> On 02/13/2009 06:44 PM, Lucio M Nicolosi wrote:
>
>> Sure it's not an elegant solution, but couldn't you just import the
>> comma delimited data to a worksheet and edit it? OpenOffice or Symphony
>> would do it.
>>
>> L.
>>
> I could, but I end up having to do something like this periodically. I
> hoped to learn a quicker way to approach it. Since I am on a deadline
> this time around, I may grep the file as I said, then use a OO-calc to
> delete the columns I don't want.
If you intend to use OO-Calc at some point and it is a repetitive task,
a spreadsheet would certainly do from the beginning, mainly if you keep
a template with the appropriate functions. 2500 lines isn't a lot of
info at all and with a Find and a Sort you would trim your databases in
seconds. But then, it wouldn't be so elegant. :-)

Regards,

L.

--
Lucio M. Nicolosi, Eng.
São Paulo - Brazil
Lat.: 23°34'4.79"S
Long.: 46°39'59.53"W

email: lmnicolosi@gmail.com
skype: lmnicolosi1

Linux Registered User #481505
http://counter.li.org/




--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

"H.S." 02-14-2009 03:24 AM

Scripting Question
 
Patton Echols wrote:
> I have a fairly massive flat file, comma delimited, that I want to
> extract info from. Specifically, I want to extract the first and last
> name and email addresses for those who have them to a new file with just
> that info. (The windows database program that this comes from simply
> will not do it) I can grep the file for the @ symbol to at least
> exclude the lines without an email address (or the @ symbol in the notes
> field) But if I can figure this out, I can also adapt what I learn for
> the next time. Can anyone point me in the right direction for my "light
> reading?"
>
> By the way, I used 'head' to get the first line, with the field names.
> This is the first of about 2300 records, the reason not to do it by hand.
>
> patton@laptop:~$ head -1 contacts.txt
> "Business Title","First Name","Middle Name","Last Name","","Business
> Company Name","","Business Title","Business Street 1","Business Street
> 2","Business Street 3","Business City","Business State","Business
> Zip","Business Country","Home Street 1","Home Street 2","Home Street
> 3","Home City","Home State","Home Zip","Home Country","Other Street
> 1","Other Street 2","Other Street 3","Other City","Other State","Other
> Zip","Other Country","Assistant Phone","Business Fax Number","Business
> Phone","Business 2 Phone","","Car Phone","","Home Fax Number","Home
> Phone","Home 2 Phone","ISDN Phone","Mobile Phone","Other Fax
> Number","Other Phone","Pager
> Phone","","","","","","","","","","","","","Busine ss Email","","Home
> Email","","Other
> Email","","","","","","","","","","","","Notes","" ,"","","","","","","","","","","","","Business
> Web Page"
>
>

Here is one crude method. Assume that the above long single line is in a
file called test.db. Then the following bash command will output the
Business Email from that file (this is one long command):
$> cat test.db | sed -e 's/(.*Business Email"),"(.*)/2/g' | awk
'BEGIN { FS = """ } ; {print $1}'

Similarly, the following gives the First name, Middle name and the Last
name.
$> cat test.db | sed -e 's/(^"Business Title"),"(.*)/2/g' | awk
'BEGIN { FS = "," } ; {print $1, $2, $3}' | tr -d '"'

Now, you can run this command on each line of your actual database file
(using the bash while and read commands) and you should get the business
email address and the names. If there is no email address, the output
will be blank.

Here is an untested set of commands to read each line from a file
(full.db) to generate names and email:
$> cat full.db | while read line; do
echo "${line}" | sed -e 's/(^"Business Title"),"(.*)/2/g' |
awk 'BEGIN { FS = "," } ; {print $1, $2, $3}' | tr -d '"';
echo "${line}" | sed -e 's/(.*Business Email"),"(.*)/2/g' |
awk 'BEGIN { FS = """ } ; {print $1}'
done

But note that this is really a crude method. I am sure others can
suggest more elegant ways to accomplish this. The above method will at
least get you started.

Warm regards.

--

Please reply to this list only. I read this list on its corresponding
newsgroup on gmane.org. Replies sent to my email address are just
filtered to a folder in my mailbox and get periodically deleted without
ever having been read.


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

"H.S." 02-14-2009 03:41 AM

Scripting Question
 
H.S. wrote:
> Patton Echols wrote:
>> I have a fairly massive flat file, comma delimited, that I want to
>> extract info from. Specifically, I want to extract the first and last
>> name and email addresses for those who have them to a new file with just
>> that info. (The windows database program that this comes from simply
>> will not do it) I can grep the file for the @ symbol to at least
>> exclude the lines without an email address (or the @ symbol in the notes
>> field) But if I can figure this out, I can also adapt what I learn for
>> the next time. Can anyone point me in the right direction for my "light
>> reading?"
>>
>> By the way, I used 'head' to get the first line, with the field names.
>> This is the first of about 2300 records, the reason not to do it by hand.
>>
>> patton@laptop:~$ head -1 contacts.txt
>> "Business Title","First Name","Middle Name","Last Name","","Business
>> Company Name","","Business Title","Business Street 1","Business Street
>> 2","Business Street 3","Business City","Business State","Business
>> Zip","Business Country","Home Street 1","Home Street 2","Home Street
>> 3","Home City","Home State","Home Zip","Home Country","Other Street
>> 1","Other Street 2","Other Street 3","Other City","Other State","Other
>> Zip","Other Country","Assistant Phone","Business Fax Number","Business
>> Phone","Business 2 Phone","","Car Phone","","Home Fax Number","Home
>> Phone","Home 2 Phone","ISDN Phone","Mobile Phone","Other Fax
>> Number","Other Phone","Pager
>> Phone","","","","","","","","","","","","","Busine ss Email","","Home
>> Email","","Other
>> Email","","","","","","","","","","","","Notes","" ,"","","","","","","","","","","","","Business
>> Web Page"
>>
>>
>
> Here is one crude method. Assume that the above long single line is in a
> file called test.db. Then the following bash command will output the
> Business Email from that file (this is one long command):
> $> cat test.db | sed -e 's/(.*Business Email"),"(.*)/2/g' | awk
> 'BEGIN { FS = """ } ; {print $1}'
>
> Similarly, the following gives the First name, Middle name and the Last
> name.
> $> cat test.db | sed -e 's/(^"Business Title"),"(.*)/2/g' | awk
> 'BEGIN { FS = "," } ; {print $1, $2, $3}' | tr -d '"'
>
> Now, you can run this command on each line of your actual database file
> (using the bash while and read commands) and you should get the business
> email address and the names. If there is no email address, the output
> will be blank.
>
> Here is an untested set of commands to read each line from a file
> (full.db) to generate names and email:
> $> cat full.db | while read line; do
> echo "${line}" | sed -e 's/(^"Business Title"),"(.*)/2/g' |
> awk 'BEGIN { FS = "," } ; {print $1, $2, $3}' | tr -d '"';
> echo "${line}" | sed -e 's/(.*Business Email"),"(.*)/2/g' |
> awk 'BEGIN { FS = """ } ; {print $1}'
> done
>
> But note that this is really a crude method. I am sure others can
> suggest more elegant ways to accomplish this. The above method will at
> least get you started.
>
> Warm regards.
>

More concise (given the order of data fields is constant) and probably
more efficient and better (the following is one long line):

#---------------------------------------------
$> cat full.db | while read line; do echo "${line}" |awk 'BEGIN { FS =
"," }; {print $2, $3, $4, $58}' | tr -d '"'; done
#---------------------------------------------

assuming your database file is called full.db

Hope that helps.
Regards,
->HS
--

Please reply to this list only. I read this list on its corresponding
newsgroup on gmane.org. Replies sent to my email address are just
filtered to a folder in my mailbox and get periodically deleted without
ever having been read.


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users


All times are GMT. The time now is 03:00 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.