FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 01-11-2008, 10:54 AM
Jeremy Sanders
 
Default Checksumming layer

Is there any sort of checksumming layer that could lie between the disk and
ext3, or be implemented as part of ext3/4?

We've just had a couple of drives recently where the drive started silently
corrupting the data without generating any I/O or SMART errors. This is
pretty disastrous as you don't necessarily find out about the corruption
until it is too late.

I imagine the overhead of such a layer wouldn't be that much. I would pay a
few percent performance for knowing that the data is not corrupt.

Jeremy

--
Jeremy Sanders <jss@ast.cam.ac.uk> http://www-xray.ast.cam.ac.uk/~jss/
X-Ray Group, Institute of Astronomy, University of Cambridge, UK.
Public Key Server PGP Key ID: E1AAE053

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 11:26 AM
"Christian Kujau"
 
Default Checksumming layer

On Fri, January 11, 2008 12:54, Jeremy Sanders wrote:
> Is there any sort of checksumming layer that could lie between the disk
> and ext3, or be implemented as part of ext3/4?

http://www.bullopensource.org/ext4/files/ext4.txt notes:

* journal checksumming for robustness, performance (prototype exists)
Features like metadata checksumming have been discussed and planned for
a bit but no patches exist yet so I'm not sure they're in the near-term
roadmap.

...but apart from that, only ZFS comes to mind:
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Checksums

C.
--
make bzImage, not war

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 11:38 AM
Jordi Prats
 
Default Checksumming layer

Hi,
You could use tripwire to check periodically all files instead of relay
on the file system for that task. (I think no file system does this
checking by now)


Jordi

Jeremy Sanders wrote:

Is there any sort of checksumming layer that could lie between the disk and
ext3, or be implemented as part of ext3/4?

We've just had a couple of drives recently where the drive started silently
corrupting the data without generating any I/O or SMART errors. This is
pretty disastrous as you don't necessarily find out about the corruption
until it is too late.

I imagine the overhead of such a layer wouldn't be that much. I would pay a
few percent performance for knowing that the data is not corrupt.

Jeremy





--
.................................................. ....................
__
/ / Jordi Prats
C E / S / C A Dept. de Sistemes
/_/ Centre de Supercomputació de Catalunya

Gran Capitą, 2-4 (Edifici Nexus) · 08034 Barcelona
T. 93 205 6464 · F. 93 205 6979 · jprats@cesca.es
.................................................. ....................


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 11:38 AM
Jordi Prats
 
Default Checksumming layer

Hi,
You could use tripwire to check periodically all files instead of relay
on the file system for that task. (I think no file system does this
checking by now)


Jordi

Jeremy Sanders wrote:

Is there any sort of checksumming layer that could lie between the disk and
ext3, or be implemented as part of ext3/4?

We've just had a couple of drives recently where the drive started silently
corrupting the data without generating any I/O or SMART errors. This is
pretty disastrous as you don't necessarily find out about the corruption
until it is too late.

I imagine the overhead of such a layer wouldn't be that much. I would pay a
few percent performance for knowing that the data is not corrupt.

Jeremy





--
.................................................. ....................
__
/ / Jordi Prats
C E / S / C A Dept. de Sistemes
/_/ Centre de Supercomputació de Catalunya

Gran Capitą, 2-4 (Edifici Nexus) · 08034 Barcelona
T. 93 205 6464 · F. 93 205 6979 · jprats@cesca.es
.................................................. ....................


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 11:44 AM
Jeremy Sanders
 
Default Checksumming layer

Jordi Prats wrote:

> You could use tripwire to check periodically all files instead of relay
> on the file system for that task. (I think no file system does this
> checking by now)

That's a possible idea.

I would have thought it would be relatively simple to write a block device
which acted a layer between the file system and real block device. I
suppose the difficultly is getting all the corner cases correct. I've never
written any kernel code, so maybe I should investigate doing that for
fun...

Jeremy

--
Jeremy Sanders <jss@ast.cam.ac.uk> http://www-xray.ast.cam.ac.uk/~jss/
X-Ray Group, Institute of Astronomy, University of Cambridge, UK.
Public Key Server PGP Key ID: E1AAE053

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 06:55 PM
tweeks
 
Default Checksumming layer

On Friday 11 January 2008 06:44, Jeremy Sanders wrote:
> Jordi Prats wrote:
> > You could use tripwire to check periodically all files instead of relay
> > on the file system for that task. (I think no file system does this
> > checking by now)
>
> That's a possible idea.
>
> I would have thought it would be relatively simple to write a block device
> which acted a layer between the file system and real block device. I
> suppose the difficultly is getting all the corner cases correct. I've never
> written any kernel code, so maybe I should investigate doing that for
> fun...

All files in the system are already hashed. You can see this by doing
an "rpm -Va". For example.. to create a baseline of a system to compare
against, just cron a script to:
rpm -Va > /root/RPMV/system-rpm-baseline.txt

then once/day or whatever, do a diff... or just grep for any "bin" directory
changes and diff that. I like this better than messing with tripwire. It's
already there, native, and easy to use.

Tweeks


Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace
Managed Hosting. Any dissemination, distribution or copying of the enclosed
material is prohibited. If you receive this transmission in error, please
notify us immediately by e-mail at abuse@rackspace.com, and delete the
original message. Your cooperation is appreciated.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 07:13 PM
Forest Bond
 
Default Checksumming layer

Hi,

On Fri, Jan 11, 2008 at 01:55:46PM -0600, tweeks wrote:
> On Friday 11 January 2008 06:44, Jeremy Sanders wrote:
> > Jordi Prats wrote:
> > > You could use tripwire to check periodically all files instead of relay
> > > on the file system for that task. (I think no file system does this
> > > checking by now)
> >
> > That's a possible idea.
> >
> > I would have thought it would be relatively simple to write a block device
> > which acted a layer between the file system and real block device. I
> > suppose the difficultly is getting all the corner cases correct. I've never
> > written any kernel code, so maybe I should investigate doing that for
> > fun...
>
> All files in the system are already hashed. You can see this by doing
> an "rpm -Va". For example.. to create a baseline of a system to compare
> against, just cron a script to:
> rpm -Va > /root/RPMV/system-rpm-baseline.txt
>
> then once/day or whatever, do a diff... or just grep for any "bin" directory
> changes and diff that. I like this better than messing with tripwire. It's
> already there, native, and easy to use.

This is specific to:

* RPM-based systems
* files provided by RPMs

Consequently, it's only useful on certain systems, and, even then, only with
certain files. That's not very good coverage, is it?

This is especially true when you consider that the files that came from the
package manager are usually the ones that you don't care about as much when
you've lost data.

-Forest
--
Forest Bond
http://www.alittletooquiet.net
_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 09:09 PM
Andreas Dilger
 
Default Checksumming layer

On Jan 11, 2008 12:44 +0000, Jeremy Sanders wrote:
> I would have thought it would be relatively simple to write a block device
> which acted a layer between the file system and real block device. I
> suppose the difficultly is getting all the corner cases correct. I've never
> written any kernel code, so maybe I should investigate doing that for
> fun...

I think at one point there was a checksumming loop driver, and adding a
checksumming mechanism to DM wouldn't be so hard either.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 01-11-2008, 09:52 PM
tweeks
 
Default Checksumming layer

On Friday 11 January 2008 14:13, Forest Bond wrote:
> Hi,
>
> On Fri, Jan 11, 2008 at 01:55:46PM -0600, tweeks wrote:
> > On Friday 11 January 2008 06:44, Jeremy Sanders wrote:
> > > Jordi Prats wrote:
> > > > You could use tripwire to check periodically all files instead of
> > > > relay on the file system for that task. (I think no file system does
> > > > this checking by now)
> > >
> > > That's a possible idea.
> > >
> > > I would have thought it would be relatively simple to write a block
> > > device which acted a layer between the file system and real block
> > > device. I suppose the difficultly is getting all the corner cases
> > > correct. I've never written any kernel code, so maybe I should
> > > investigate doing that for fun...
> >
> > All files in the system are already hashed. You can see this by doing
> > an "rpm -Va". For example.. to create a baseline of a system to compare
> > against, just cron a script to:
> > rpm -Va > /root/RPMV/system-rpm-baseline.txt
> >
> > then once/day or whatever, do a diff... or just grep for any "bin"
> > directory changes and diff that. I like this better than messing with
> > tripwire. It's already there, native, and easy to use.
>
> This is specific to:
>
> * RPM-based systems
> * files provided by RPMs
> Consequently, it's only useful on certain systems,

Heh.. well.. last I checked, this is a redhat ext3 list. Red hat uses rpm..
and no one but Red hat still actually uses ext3 right? (hehe)...

> and, even then, only
> with certain files. That's not very good coverage, is it?

Uhh.. all SYSTEM files.. which is all I'm looking at when doing compromise
checks (except for root kits, etc.. for which I use separate tools).


> This is especially true when you consider that the files that came from the
> package manager are usually the ones that you don't care about as much when
> you've lost data.

You tripwire scan data files? Hmm..

I've seen hundred of compromised servers... 80-90% of them can be detected
with a simple RPM scan. The ones you can't are the ones where hacks have
deleted the RPM DBs. but in that case, your baseline diff sets off red flags
anyway. It's actually a pretty good scan to run nightly/weekly, etc (along
with root kit scans, etc). In fact.. I prefer using unorthodox detection
methods rather than well known forms of F.A.M. (file alteration monitoring)
like tripwire which if seen, are instantly attacked and disabled.

Tweeks


Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace
Managed Hosting. Any dissemination, distribution or copying of the enclosed
material is prohibited. If you receive this transmission in error, please
notify us immediately by e-mail at abuse@rackspace.com, and delete the
original message. Your cooperation is appreciated.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 07:41 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org