FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 02-20-2010, 03:15 AM
Reza Roboubi
 
Default Is my data checksummed?

What checksumming is done for the actual data? I know that storage
devices often do their own checksumming too, but how can I be sure my
data is integrity checked every time I read it?


PS:
ext2/ext4?

Thank you very much.

Reza.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-20-2010, 11:42 PM
Christian Kujau
 
Default Is my data checksummed?

On Fri, 19 Feb 2010 at 20:15, Reza Roboubi wrote:
> What checksumming is done for the actual data?

Ext4 introduced journal checksumming[0], but not for the actual data.
Ext2/3 don't have checksums at all, IIRC.

> often do their own checksumming too, but how can I be sure my data is
> integrity checked every time I read it?

Btrfs has data checksumming :-)

Christian.

[0] http://ext4.wiki.kernel.org/index.php/Ext4_Howto#Journal_checksumming
--
BOFH excuse #144:

Too few computrons available.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-22-2010, 01:41 AM
 
Default Is my data checksummed?

On Fri, Feb 19, 2010 at 08:15:30PM -0800, Reza Roboubi wrote:
> What checksumming is done for the actual data? I know that storage
> devices often do their own checksumming too, but how can I be sure
> my data is integrity checked every time I read it?

If you use disks that support the Data Integrity Field (DIF)
extension, Linux will use it to provide end-to-end data checksum
support. Otherwise, there are checksums on the disk and between disk
controller and the CPU, but those are obviously not end-to-end
checksums.

Adding data-level checksums is not something that we are planning on
adding to the ext2/3/4 file systems. BTRFS is the only file system
that has data-level checksums, but it's not yet production ready.

Best regards,

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-22-2010, 11:49 AM
Ric Wheeler
 
Default Is my data checksummed?

On 02/21/2010 09:41 PM, tytso@mit.edu wrote:

On Fri, Feb 19, 2010 at 08:15:30PM -0800, Reza Roboubi wrote:


What checksumming is done for the actual data? I know that storage
devices often do their own checksumming too, but how can I be sure
my data is integrity checked every time I read it?


If you use disks that support the Data Integrity Field (DIF)
extension, Linux will use it to provide end-to-end data checksum
support. Otherwise, there are checksums on the disk and between disk
controller and the CPU, but those are obviously not end-to-end
checksums.



Just to be clear, even with a storage path that supports DIF/DIX, we
don't currently do anything for applications on top of file systems. The
primary application to target storage path is covered mainly for raw
devices.


ric


Adding data-level checksums is not something that we are planning on
adding to the ext2/3/4 file systems. BTRFS is the only file system
that has data-level checksums, but it's not yet production ready.

Best regards,

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users



_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-23-2010, 08:56 AM
Reza Roboubi
 
Default Is my data checksummed?

Thanks a lot for all the responses.

Ric Wheeler wrote:

On 02/21/2010 09:41 PM, tytso@mit.edu wrote:

If you use disks that support the Data Integrity Field (DIF)
extension, Linux will use it to provide end-to-end data checksum
support. Otherwise, there are checksums on the disk and between disk
controller and the CPU, but those are obviously not end-to-end
checksums.



Just to be clear, even with a storage path that supports DIF/DIX, we
don't currently do anything for applications on top of file systems. The
primary application to target storage path is covered mainly for raw
devices.


Sorry Ric,

The last sentence loses me. I mean, I know what raw devices are!
Would you please elaborate a little.

Reza.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-23-2010, 10:46 PM
Ric Wheeler
 
Default Is my data checksummed?

On 02/23/2010 04:56 AM, Reza Roboubi wrote:

Thanks a lot for all the responses.

Ric Wheeler wrote:

On 02/21/2010 09:41 PM, tytso@mit.edu wrote:

If you use disks that support the Data Integrity Field (DIF)
extension, Linux will use it to provide end-to-end data checksum
support. Otherwise, there are checksums on the disk and between disk
controller and the CPU, but those are obviously not end-to-end
checksums.


Just to be clear, even with a storage path that supports DIF/DIX, we
don't currently do anything for applications on top of file systems.
The primary application to target storage path is covered mainly for
raw devices.


Sorry Ric,

The last sentence loses me. I mean, I know what raw devices are!
Would you please elaborate a little.

Reza.


Sorry for the confusion. It confuses me now that I try to reread it as well :-(

What I was trying to say is that in this first implementation, the only
applications that can take advantage of DIF/DIX extra data protection will need
to do it all themselves and will need to use raw devices (no file systems).


Oracle's DB will be a likely candidate as will a very few other sophisticated
applications.


Note that the other mode of operation (HBA -> target storage device) is
invisible and will work for everyone. It just does not protect against data
corruption higher in the stack (above the HBA/device driver).


ric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 02-24-2010, 03:38 PM
 
Default Is my data checksummed?

>>> What checksumming is done for the actual data? I know that
>>> storage devices often do their own checksumming too, but how
>>> can I be sure my data is integrity checked every time I read
>>> it?

These things ("storage devices often do their own checksumming"
and "my data is integrity checked") are rather unrelated.

Various parts of storage subsystems do things like checksumming
not to protect your data, but to detect potential faults. That
is mainly as a diagnostic not for integrity.

Part of the reason is that it is very difficult and needlessly
expensive to do comprehensivce integrity checking within the
storage subsystem, automagically.

>> If you use disks that support the Data Integrity Field (DIF)
>> extension, Linux will use it to provide end-to-end data
>> checksum support. Otherwise, there are checksums on the disk
>> and between disk controller and the CPU, but those are
>> obviously not end-to-end checksums.

Yes. But I'll add that the only way to ensure that "data is
integrity checked" is to do it truly end-to-end, with data and
application specific checks. For example as a weak but useful
measure I 'zip' or gzip' (sometimes with zero compression if
already compressed) data that I want to be able to move around
across years and many storage devices.

Consider for example bugs in the IO subsystem itself, where the
wrong data ends up being written and checksummed, and gets
validated every time even if it is not the right data.

> Just to be clear, even with a storage path that supports
> DIF/DIX, we don't currently do anything for applications on
> top of file systems. The primary application to target storage
> path is covered mainly for raw devices.

Which makes it not that generally useful. In effect DIF is a hw
accelerator of a weak form of per-block checksumming. I think
that most current CPUs are fast enough to do it without it
beoing that noticeable.

>> Adding data-level checksums is not something that we are
>> planning on adding to the ext2/3/4 file systems. BTRFS is
>> the only file system that has data-level checksums, but it's
>> not yet production ready.

But again that's not end-to-end. It is just as far as the
current storage system goes, and the biggest value, like for
ZFS, is to detect issues with the storage system itself (e.g.
bugs as well as hw issues).

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 07:41 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org