FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 05-04-2011, 04:17 PM
"Nicolas Ross"
 
Default Finding wich files a writen to

Hi !

I have a server (Centos 5) that is using a pair of SAS drives to store the
data. (Mail server) They are on an adaptec raid controler with a battery
backup and write back cache active.

>From time to time, I have sever peak io to those data disks (> 400 to 500
iops, > 70 to 100 megs/sec).

With iostat, I find that it's almost a write i/o problem. How can I find to
which files the OS writes ? On OSX boxes, there is a utility called fs_usage
that can reports any disk activity for a particular process or all
processes. Is there any utility like this on Centos ?

iotop can points me to wich process, but that doesn't points me to what
files are the culprits...

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-04-2011, 06:01 PM
Jorge Fábregas
 
Default Finding wich files a writen to

On 05/04/2011 12:17 PM, Nicolas Ross wrote:
> iotop can points me to wich process, but that doesn't points me to what
> files are the culprits...

A rough way would be to change to the top-level directory where you
suspect the files are being written and perform:

find . -type f -mmin -1 (that would search for all files modified
within the last minute)

A more elegant way would be:

lsof -p PID (where PID is the process ID...of the process iotop showed you)

HTH,
Jorge
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 01:01 AM
Marian Marinov
 
Default Finding wich files a writen to

On Wednesday 04 May 2011 21:01:03 Jorge Fábregas wrote:
> On 05/04/2011 12:17 PM, Nicolas Ross wrote:
> > iotop can points me to wich process, but that doesn't points me to what
> > files are the culprits...
>
> A rough way would be to change to the top-level directory where you
> suspect the files are being written and perform:
>
> find . -type f -mmin -1 (that would search for all files modified
> within the last minute)
>
> A more elegant way would be:
>
> lsof -p PID (where PID is the process ID...of the process iotop showed
> you)
>

Just out of curiosity I decided to write a simple script which checks all the
files from all pids on the system.

Here is what I got:
http://hydra.azilian.net/scripts/read_fds.pl

The idea is to read all the /proc/PID/fdinfo/ files and check the difference in
the pos lines (the position in the file descriptor). This is both write and
read position depending on how the application has opened the file.
So in the end it lists all pids and the respective FDs which have changes:

hackman@gamelon:~$ sudo ./read_fds.pl 4
Pid: 14229 Position change: 22 blocks FD: 4(/home/hackman/f2.tst)
Pid: 14229 Position change: 12 blocks FD: 3(/home/hackman/f1.tst)

The argument to the script is the sleep between the two checks.
I have tested the script on a few production servers... It works as a charm

Thank you for the good question... now I have one good tool in my arsenal

--
Best regards,
Marian Marinov
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:02 AM
Nicolas Ross
 
Default Finding wich files a writen to

> Just out of curiosity I decided to write a simple script which checks all the
> files from all pids on the system.
>
> Here is what I got:
> http://hydra.azilian.net/scripts/read_fds.pl
>
> The idea is to read all the /proc/PID/fdinfo/ files and check the difference in
> the pos lines (the position in the file descriptor). This is both write and
> read position depending on how the application has opened the file.
> So in the end it lists all pids and the respective FDs which have changes:
>
> hackman@gamelon:~$ sudo ./read_fds.pl 4
> Pid: 14229 Position change: 22 blocks FD: 4(/home/hackman/f2.tst)
> Pid: 14229 Position change: 12 blocks FD: 3(/home/hackman/f1.tst)
>
> The argument to the script is the sleep between the two checks.
> I have tested the script on a few production servers... It works as a charm
>
> Thank you for the good question... now I have one good tool in my arsenal
This is excellent, and sooo clever... Except that I don't have the
/proc/*/fdinfo directories. It seems that theses directories appeared in
2.6.22, and, since I am in centos5, I only have 2.6.18...

I tested it on SL6 machine, and it works perfectly... Upgrade is not an
option for the moment for the machine I have the problem with.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:24 AM
Marcelo Beckmann
 
Default Finding wich files a writen to

2011/5/4 Nicolas Ross <rossnick-lists@cybercat.ca>:
> Hi !
>
> I have a server (Centos 5) that is using a pair of SAS drives to store the
> data. (Mail server) They are on an adaptec raid controler with a battery
> backup and write back cache active.
>
> >From time to time, I have sever peak io to those data disks (> 400 to 500
> iops, > 70 to 100 megs/sec).
>
> With iostat, I find that it's almost a write i/o problem. How can I find to
> which files the OS writes ? On OSX boxes, there is a utility called fs_usage
> that can reports any disk activity for a particular process or all
> processes. Is there any utility like this on Centos ?
>
> iotop can points me to wich process, but that doesn't points me to what
> files are the culprits...

I sugest a look for tools like this
http://freshmeat.net/projects/fsniper

it helps to make a script to watch file activities, and it uses a kernel feature

I discovered inotify some months ago when I looked into every
initscript in init.d

[23:13:35 root@gw init.d]# cat /etc/redhat-release
CentOS release 5.3 (Final)
[23:13:45 root@gw init.d]# head restorecond
#!/bin/sh
#
# restorecond: Daemon used to maintain path file context
#
# chkconfig: 2345 12 87
# description: restorecond uses inotify to look for creation of new files
# listed in the /etc/selinux/restorecond.conf file, and restores the
# correct security context.


more about inotify:
http://linux.die.net/man/7/inotify

http://www.linuxjournal.com/article/8478
What Is inotify?

inotify is a file change notification system—a kernel feature that
allows applications to request the monitoring of a set of files
against a list of events. When the event occurs, the application is
notified. To be useful, such a feature must be simple to use,
lightweight with little overhead and flexible. It should be easy to
add new watches and painless to receive notification of events.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:34 AM
"Nicolas Ross"
 
Default Finding wich files a writen to

(...)

> This is excellent, and sooo clever... Except that I don't have the
> /proc/*/fdinfo directories. It seems that theses directories appeared in
> 2.6.22, and, since I am in centos5, I only have 2.6.18...
>
> I tested it on SL6 machine, and it works perfectly... Upgrade is not an
> option for the moment for the machine I have the problem with.

It appears that RHEL has back-ported some featears in 2.6.18 kernels. My
service is now at 5.6 (was 5.5). Now I have the /proc/*/fdinfo directories,
and can use your tool.

Best regards,

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 02:40 AM
Marian Marinov
 
Default Finding wich files a writen to

On Thursday 05 May 2011 05:24:10 Marcelo Beckmann wrote:
> 2011/5/4 Nicolas Ross <rossnick-lists@cybercat.ca>:
> > Hi !
> >
> > I have a server (Centos 5) that is using a pair of SAS drives to store
> > the data. (Mail server) They are on an adaptec raid controler with a
> > battery backup and write back cache active.
> >
> > >From time to time, I have sever peak io to those data disks (> 400 to
> > >500
> >
> > iops, > 70 to 100 megs/sec).
> >
> > With iostat, I find that it's almost a write i/o problem. How can I find
> > to which files the OS writes ? On OSX boxes, there is a utility called
> > fs_usage that can reports any disk activity for a particular process or
> > all processes. Is there any utility like this on Centos ?
> >
> > iotop can points me to wich process, but that doesn't points me to what
> > files are the culprits...
>
> I sugest a look for tools like this
> http://freshmeat.net/projects/fsniper
>
> it helps to make a script to watch file activities, and it uses a kernel
> feature
>
> I discovered inotify some months ago when I looked into every
> initscript in init.d
>
> [23:13:35 root@gw init.d]# cat /etc/redhat-release
> CentOS release 5.3 (Final)
> [23:13:45 root@gw init.d]# head restorecond
> #!/bin/sh
> #
> # restorecond: Daemon used to maintain path file context
> #
> # chkconfig: 2345 12 87
> # description: restorecond uses inotify to look for creation of new files
> # listed in the /etc/selinux/restorecond.conf file, and restores the #
> correct security context.
>
>
> more about inotify:
> http://linux.die.net/man/7/inotify
>
> http://www.linuxjournal.com/article/8478
> What Is inotify?
>
> inotify is a file change notification system—a kernel feature that
> allows applications to request the monitoring of a set of files
> against a list of events. When the event occurs, the application is
> notified. To be useful, such a feature must be simple to use,
> lightweight with little overhead and flexible. It should be easy to
> add new watches and painless to receive notification of events.

If you go the inotify route, do keep in mind that you need to monitor for
modify events, otherwise you would not see the file changes before the
applications finish with the files.

Regards,
Marian


--
Best regards,
Marian Marinov
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-05-2011, 08:51 AM
 
Default Finding wich files a writen to

On Wed, May 04, 2011 at 12:17:15PM -0400, Nicolas Ross wrote:
> Hi !
>
> I have a server (Centos 5) that is using a pair of SAS drives to store the
> data. (Mail server) They are on an adaptec raid controler with a battery
> backup and write back cache active.
>
> >From time to time, I have sever peak io to those data disks (> 400 to 500
> iops, > 70 to 100 megs/sec).
>
> With iostat, I find that it's almost a write i/o problem. How can I find to
> which files the OS writes ? On OSX boxes, there is a utility called fs_usage
> that can reports any disk activity for a particular process or all
> processes. Is there any utility like this on Centos ?
>
> iotop can points me to wich process, but that doesn't points me to what
> files are the culprits...

Systemtap can[*] be very useful for this.
[*] I use DTrace under Solaris. This is one of the best OS feature any sysadmin can have.
Systemtap is similar to DTrace (at least it tries to be ...).
Look at http://uselessuseofcat.com/?p=281


Regards
Przemyslaw Bak (przemol)



















































-----------------------------------------
Wez udzial w konkursie i WYGRAJ!
Sprawdz >> http://linkint.pl/f299e

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2011, 09:55 AM
Rajagopal Swaminathan
 
Default Finding wich files a writen to

Greetings,

On Thu, May 5, 2011 at 2:21 PM, <przemolicc@poczta.fm> wrote:
> On Wed, May 04, 2011 at 12:17:15PM -0400, Nicolas Ross wrote:
>> Hi !
>>
>> With iostat, I find that it's almost a write i/o problem. How can I find to
>> which files the OS writes ? On OSX boxes, there is a utility called fs_usage
>> that can reports any disk activity for a particular process or all
>> processes. Is there any utility like this on Centos ?

dunno if inotify et al helps your cause


--
Regards,

Rajagopal
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 12:05 PM
John Doe
 
Default Finding wich files a writen to

From: Nicolas Ross <rossnick-lists@cybercat.ca>

> With iostat, I find that it's almost a write i/o problem. How can I find to
> which files the OS writes ? On OSX boxes, there is a utility called fs_usage
> that can reports any disk activity for a particular process or all
> processes. Is there any utility like this on Centos ?
> iotop can points me to wich process, but that doesn't points me to what
> files are the culprits...

Maybe the following would give some info...?
lsof | grep "[0-9]w " | grep " /"

JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 03:53 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org