FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 01-03-2008, 12:09 PM
"William L. Maltby"
 
Default Random files in homedir gets deleted

On Thu, 2008-01-03 at 13:09 +0100, Christopher Thorjussen wrote:
> On one of my systems I seem to loose a file or two from time to time.
> Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was
> deleted/removed/vanished. Another time it was /home/online/sh/daemon
> that was deleted.
>
> But I can't seem to find anything strange in the logs or in the history,
> nor would any of my scripts running in crontab mess with those files.
>
> Where can I look for clues? And how do I enable audit for file
> operations in my home folder?

Although less common than "In The Day", if file system corruption was
repaired by *fsck during boot or other times, the "lost+found" directory
of the affected FS may contain some files with weird names, having
mostly numbers. These numbers are i-node numbers associated with the
"chunks" that appear in the lost+found directory.

If you see these, it indicates that corruption occurred and was
repaired. If the files are plain text, there is a possibility to
reconstruct the original files via visual inspection and concatenation.
Often they are mixed or binary and more difficult to reconstruct.

If it's not corruption, finding the cause may not be so easy. Long ago,
before the we had the plethora of tools now available, I found the cause
of such a mystery with a multi-pronged approach.

1. Fire up the system accounting package (sar).
2. Start a "daemon" (really just a "nohup <your shell script name> &"
that checked for the existence of the file every so many seconds.
This if really nothing more than a

if [ ! -r <file name> ] ; then <some commands> ; fi

The "-r" can be replaced/compounded with other flags as desired. See
"man bash", "CONDITIONAL EXPRESSIONS.

When the file went missing, a "log" entry was generated.

3. Generate the sar reports and examine them to see what was running
around that time.

4. As I did, go viciously beat the offending user that made a flawed
script about the ears with a rather rigid clue bat.

5. Problem solved.

>
>
> /Christopher Thorjussen
> <snip sig stuff>

HTH
--
Bill

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-03-2008, 12:44 PM
"Ross S. W. Walker"
 
Default Random files in homedir gets deleted

Title: Re: [CentOS] Random files in homedir gets deleted









You can enable auditing to determine if the files are disappearing due to human/machine intervention (audit file system deletes) or if it is due to file system corruption (files disappear and no delete audits recorded).



It may just be an errant rsync script.





-Ross





-----Original Message-----

From: centos-bounces@centos.org <centos-bounces@centos.org>

To: centos@centos.org <centos@centos.org>

Sent: Thu Jan 03 07:09:11 2008

Subject: [CentOS] Random files in homedir gets deleted



On one of my systems I seem to loose a file or two from time to time.

Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was

deleted/removed/vanished. Another time it was /home/online/sh/daemon

that was deleted.



But I can't seem to find anything strange in the logs or in the history,

nor would any of my scripts running in crontab mess with those files.



Where can I look for clues? And how do I enable audit for file

operations in my home folder?





/Christopher Thorjussen



_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos










This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-03-2008, 12:59 PM
 
Default Random files in homedir gets deleted

>
> On one of my systems I seem to loose a file or two from time
> to time. Last night, one of my files
> (/home/online/sh/NattjobbPrivat.sh) was
> deleted/removed/vanished. Another time it was
> /home/online/sh/daemon that was deleted.
>
> But I can't seem to find anything strange in the logs or in
> the history, nor would any of my scripts running in crontab
> mess with those files.
>
> Where can I look for clues? And how do I enable audit for
> file operations in my home folder?
>


If your system is capable, use the SMART tools to check your drive out
(as CM suggests), something like this:

smartctl -a /dev/sda
<replace /dev/sda with the drive in question>
See how your 'error count log' is doing. If there are errors, then you
might want to run that command a few times and see if the error count is
still rising.


Is it everything in the /home/online/sh/ directory that is getting
deleted, or can you see any pattern at all? (it sounds like it is random
from what you said...but hard to think of why files would be deleted
randomly....as you know!)
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-03-2008, 01:09 PM
Fajar Priyanto
 
Default Random files in homedir gets deleted

On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote:
> On one of my systems I seem to loose a file or two from time to time.
> Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was
> deleted/removed/vanished. Another time it was /home/online/sh/daemon
> that was deleted.
>
> But I can't seem to find anything strange in the logs or in the history,
> nor would any of my scripts running in crontab mess with those files.
>
> Where can I look for clues? And how do I enable audit for file
> operations in my home folder?

Hi, this really sounds weird. In order to audit it, the following checklist
might help:
1. If the system was administered by an admin other than you and he got
fired/dismissed with hard feeling on him, he might put a crontab that would
do nasty thing randomly. Audit all the files in:
/var/spool/cron
/var/spool/at
Also all the script in /etc/cron.{d,daily,weekly,monthly}, /etc/crontab

2, Audit all RPM files installed using:
rpm -Va, looks for a difference in md5sum for binary files such
as /bin/ls,/bin/ps, etc. You might want to use cracker detection script such
as rkhunter.

3. Looks for the word "error" in log files:
grep -r error /var/log
See for related error such as filesystem corruption, etc

4. It's a long shot, but could be a misconfigured rsync script?

HTH, pls let us know the result.
--
Fajar Priyanto | Reg'd Linux User #327841 | Linux tutorial
http://linux2.arinet.org
21:09:01 up 1:02, 2.6.22-14-generic GNU/Linux
Let's use OpenOffice. http://www.openoffice.org
The real challenge of teaching is getting your students motivated to learn.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 07:20 AM
"Christopher Thorjussen"
 
Default Random files in homedir gets deleted

>On Thu, 3 Jan 2008 13:09:11 +0100
>"Christopher Thorjussen" <Christopher.Thorjussen@carrot.no> wrote:
>
>> On one of my systems I seem to loose a file or two from time to time.
>>
>> Where can I look for clues?
>Is your system visible to the internet? Maybe it's running some kind of
>Apache with homedirs loosely enabled and one unsecure php script + one
>little h4x0r could do the trick? Is your file system sane? Is your
hard
>drive(s) SMART-wise OK?
>CM

The system is visible only for a few defined IP addresses on the
internet

Apache is not installed. It runs Oracle 10.2g

/Christopher
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 07:46 AM
"Christopher Thorjussen"
 
Default Random files in homedir gets deleted

>> Where can I look for clues? And how do I enable audit for
>> file operations in my home folder?
>>
>
> If your system is capable, use the SMART tools to check your drive out
> (as CM suggests), something like this:
>
> smartctl -a /dev/sda
> <replace /dev/sda with the drive in question>
> See how your 'error count log' is doing. If there are errors, then you
> might want to run that command a few times and see if the error count
is
> still rising.

It's a Dell PowerEdge 2950 running in raid 1+0 on the Perc 5/I with SCSI
drives.

[root@ora01 ~]# smartctl -a /dev/sda7
smartctl version 5.33 [x86_64-redhat-linux-gnu] Copyright (C) 2002-4
Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: DELL PERC 5/i Version: 1.03
Serial number: 008f71137876e77c0e00b4fdc230c201
Device type: disk
Local Time is: Fri Jan 4 09:43:37 2008 CET
Device does not support SMART

Error Counter logging not supported

Error Events logging not supported
Device does not support Self Test logging


> Is it everything in the /home/online/sh/ directory that is getting
> deleted, or can you see any pattern at all? (it sounds like it is
random
> from what you said...but hard to think of why files would be deleted
> randomly....as you know!)

No pattern so far. Yeah I know it sounds strange for files to be
randomly deleted.

/Christopher
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 08:23 AM
"Christopher Thorjussen"
 
Default Random files in homedir gets deleted

> On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote:
> > On one of my systems I seem to loose a file or two from time to
time.
> > Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was
> > deleted/removed/vanished. Another time it was /home/online/sh/daemon
> > that was deleted.
> >
> > But I can't seem to find anything strange in the logs or in the
history,
> > nor would any of my scripts running in crontab mess with those
files.
> >
> > Where can I look for clues? And how do I enable audit for file
> > operations in my home folder?
>
> Hi, this really sounds weird. In order to audit it, the following
> checklist
> might help:
> 1. If the system was administered by an admin other than you and he
got
> fired/dismissed with hard feeling on him, he might put a crontab that
> would
> do nasty thing randomly. Audit all the files in:
> /var/spool/cron
> /var/spool/at
> Also all the script in /etc/cron.{d,daily,weekly,monthly},
/etc/crontab

No admin or anyone else with access have quit or been fired. The files
and folders looks fine.

> 2, Audit all RPM files installed using:
> rpm -Va, looks for a difference in md5sum for binary files such
> as /bin/ls,/bin/ps, etc. You might want to use cracker detection
script
> such as rkhunter.
The files look fine. Some files are marked as MD5 mismatch but it's
mostly config files I've changed. The only files I'm not sure of is:

SM5....T /usr/share/rhn/rhn_applet/rhn_applet.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_animation.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_apt.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_dialogs.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_model.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_protocols.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpc.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpm.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_version.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_applet_yum.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_sources.pyc
SM5....T /usr/share/rhn/rhn_applet/rhn_utils.pyc

But I'm not running X so the applet isn't running.

>
> 3. Looks for the word "error" in log files:
> grep -r error /var/log
> See for related error such as filesystem corruption, etc
[root@ora01 tmp]# grep -r error /var/log
/var/log/Xorg.0.log: (WW) warning, (EE) error, (NI) not implemented,
(??) unknown.
/var/log/anaconda.log:* getting rpm error class
/var/log/prelink.log:/usr/lib64/libgpg-error.so.0.1.3
0000003c50e00000-0000003c50f02878
/var/log/rpmpkgs.4:libgpg-error-1.0-1.x86_64.rpm
/var/log/rpmpkgs.1:libgpg-error-1.0-1.x86_64.rpm
/var/log/messages.2ec 17 08:13:10 ora01 kernel: daemon[1562]: segfault
at 0000007fc0000000 rip 0000002a957af4b2 rsp 0000007fbfffe730 error 6
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/scrollkeeper.log:I/O error : Attempt to load network entity
http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep
er-omf.dtd
/var/log/rpmpkgs.2:libgpg-error-1.0-1.x86_64.rpm
/var/log/Xorg.0.log.old: (WW) warning, (EE) error, (NI) not
implemented, (??) unknown.
/var/log/rpmpkgs.3:libgpg-error-1.0-1.x86_64.rpm
/var/log/rpmpkgs:libgpg-error-1.0-1.x86_64.rpm
/var/log/anaconda.xlog: (WW) warning, (EE) error, (NI) not implemented,
(??) unknown.
/var/log/anaconda.xlog:error opening security policy file
/etc/X11/xserver/SecurityPolicy


> 4. It's a long shot, but could be a misconfigured rsync script?
Rsync is not running/used, but some custom scripts are running cleaning
up some folders. I'm trying to battle through them to see if somethings
wrong in them, but so far I've found nothing.

> HTH, pls let us know the result.
Will do.

/Christopher
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 08:25 AM
"Christopher Thorjussen"
 
Default Random files in homedir gets deleted

> You can enable auditing to determine if the files are disappearing due
to human/machine intervention (audit file system deletes) or if it is
due to file system corruption (files disappear and no delete audits
recorded).
>
> It may just be an errant rsync script.
>
> -Ross

How do I enable auditing of the home dir?

/Christopher
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 09:18 AM
"Radu Radutiu"
 
Default Random files in homedir gets deleted

Hi you can try to use the kernel audit facility:

1) enable the auditd daemon:

service auditd start

2) enable audit for the home directory (only audit write operations to
the directory inode); the command is not recursive and you cannot use
wildcards

auditctl -w /home/user -pw

3) after a file disapears use ausearch to find who removed it (and
what command was used to remove it); suppose file "test" was removed

ausearch -f /home/user/test

Radu

On Jan 4, 2008 11:25 AM, Christopher Thorjussen
<Christopher.Thorjussen@carrot.no> wrote:
>
> > You can enable auditing to determine if the files are disappearing due
> to human/machine intervention (audit file system deletes) or if it is
> due to file system corruption (files disappear and no delete audits
> recorded).
> >
> > It may just be an errant rsync script.
> >
> > -Ross
>
> How do I enable auditing of the home dir?
>
> /Christopher
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-04-2008, 12:34 PM
"Ross S. W. Walker"
 
Default Random files in homedir gets deleted

Title: Re: [CentOS] Random files in homedir gets deleted









Do what I do when I need to setup a new Linux facility.



Google "linux audit"



I remember getting a good hit near the top with that. There are cli tools for adding files/folders/mounts to the audit system and you can tailor which type of activity to audit. It's no where as difficult to do as it sounds.



-Ross





-----Original Message-----

From: centos-bounces@centos.org <centos-bounces@centos.org>

To: CentOS mailing list <centos@centos.org>

Sent: Fri Jan 04 04:25:17 2008

Subject: RE: [CentOS] Random files in homedir gets deleted





> You can enable auditing to determine if the files are disappearing due

to human/machine intervention (audit file system deletes) or if it is

due to file system corruption (files disappear and no delete audits

recorded).

>

> It may just be an errant rsync script.

>

> -Ross



How do I enable auditing of the home dir?



/Christopher

_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos










This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 12:33 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org