FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 10-29-2011, 11:19 AM
Mr Dash Four
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

As part of my livecd data structure dmsetup is used to create a snapshot
containing xz-compressed root combined with copy-on-write ram-based
overlay.


The way as I understand it, the ram-based overlay contains only the
"chunks" of data which is changed. Is there a way - in terms of a tool
or by any other means - to look at this overlay to see what data has
been changed exactly?


I have tried, rather naively as it turned out, to mount the ram-based
overlay but received "unknown filesystem type 'DM_snapshot_cow'" error
message, which makes me thing that the data structure is unique somehow
and private to the device mapper.


One other thing - my dmsetup man page is from 2006 (created some 5 years
ago) even though I am using the latest device-mapper package (1.02.63 I
believe) and this man page does not contain any information on dmsetup
create snapshot format, which is very annoying. Is there an updated
version of the man page I could look at?


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 10-29-2011, 08:49 PM
Alasdair G Kergon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

markmc did some code that shows how to read the format a few years ago here:
http://people.gnome.org/~markmc/code/merge-dm-snapshot.c

Otherwise look at dm-snap-persistent.c in the kernel tree.

Alasdair

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-27-2011, 09:31 PM
Frederick Grose
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com> wrote:


markmc did some code that shows how to read the format a few years ago here:

http://people.gnome.org/~markmc/code/merge-dm-snapshot.c



Otherwise look at dm-snap-persistent.c in the kernel tree.



Alasdair
Users can easily exhaust a LiveUSB snapshot overlay by writing


too many changes to the OS filesystem, such as by performing
a yum update.

Would such an 'exhausted' overlay be amenable to some sort of
data recovery or forensics?

If so, how so; if not, why not?



(or suitable references)

Thanks,******** --Fred


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-28-2011, 12:08 AM
Douglas McClendon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On 11/27/2011 04:31 PM, Frederick Grose wrote:

On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com
<mailto:agk@redhat.com>> wrote:

markmc did some code that shows how to read the format a few years
ago here:
http://people.gnome.org/~markmc/code/merge-dm-snapshot.c
<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>

Otherwise look at dm-snap-persistent.c in the kernel tree.

Alasdair


Users can easily exhaust a LiveUSB snapshot overlay by writing
too many changes to the OS filesystem, such as by performing
a yum update.

Would such an 'exhausted' overlay be amenable to some sort of
data recovery or forensics?

If so, how so; if not, why not?


Can you not mount the exhausted dm snapshot? If you mean data recovery
or forensics beyond that, please elaborate.


I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems
before (on actually read-only block devices). Maybe some flags that
I've forgotten get past that, but worst case you can always fake a
writable device with a 2nd overlay/snapshot going to a writable device,
i.e. if you wanted to let fsck repair things.


-dmc

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-29-2011, 05:22 AM
Frederick Grose
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon <dmc.lists@filteredperception.org> wrote:


On 11/27/2011 04:31 PM, Frederick Grose wrote:


On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon <agk@redhat.com

<mailto:agk@redhat.com>> wrote:



* *markmc did some code that shows how to read the format a few years

* *ago here:

* *http://people.gnome.org/~markmc/code/merge-dm-snapshot.c

* *<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>



* *Otherwise look at dm-snap-persistent.c in the kernel tree.



* *Alasdair





Users can easily exhaust a LiveUSB snapshot overlay by writing

too many changes to the OS filesystem, such as by performing

a yum update.



Would such an 'exhausted' overlay be amenable to some sort of

data recovery or forensics?



If so, how so; if not, why not?




Can you not mount the exhausted dm snapshot? *If you mean data recovery or forensics beyond that, please elaborate.



I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems before (on actually read-only block devices). *Maybe some flags that I've forgotten get past that, but worst case you can always fake a writable device with a 2nd overlay/snapshot going to a writable device, i.e. if you wanted to let fsck repair things.





-dmc
A typical scenario goes like this:
A LiveCD is loaded onto a USB device with a relatively small

persistent overlay and no separate home.img filesystem. *The userproceeds to use the system and save some information. *They may

execute a df command and see that there is lots of spaceavailable on the rootfs. *Then they decide to execute a yum

update or yum install when there is insufficient space available.(They do not know about dmsetup status.) *Their session will now

not shutdown normally. dmsetup status now shows
live-rw: 0 8388608 snapshot Invalid.


The device fails to complete a subsequent boot attempt. *Thestartup log shows


mount: /dev/mapper/live-rw: can't read superblock


e2fsck /dev/dm-0 shows
Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)

e2fsck: Attempt to read block from the filesystem resulted inshort read while trying to open /dev/dm-0

Could this be a zero-length partition?
I recreated the above scenario on Fedora-16-Live-Desktop

installed with an overlay-size-mb of 100. *After updating to ~70%of the overlay, I wrote with

dd if=/dev/zero of=/boot/load bs=1M count=103 times to exhaust the overlay.


With the defective device mounted on a good system, and loopdevices set up for ext3fs.img and the overlay file,


dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"reports

item: 0 8388608 snapshot Invalid
e2fsck /dev/dm-0 reports


e2fsck 1.41.14 (22-Dec-2010)e2fsck: Attempt to read block from filesystem resulted in short

read while trying to open /dev/dm-0Could this be a zero-length partition?
A hex editor view of the overlay file shows lots of content with

lots of zeros at the end of the file.
My questions are these:


Might there be a way to edit the overlay*file to restore its

utility for the information written before*the damaging write?


If so, how so; if not, why not?
Suitable references would do as well.


Thanks! * * * --Fred




--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-30-2011, 01:09 AM
Douglas McClendon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On 11/29/2011 12:22 AM, Frederick Grose wrote:

On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon
<dmc.lists@filteredperception.org
<mailto:dmc.lists@filteredperception.org>> wrote:

On 11/27/2011 04:31 PM, Frederick Grose wrote:

On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon
<agk@redhat.com <mailto:agk@redhat.com>
<mailto:agk@redhat.com <mailto:agk@redhat.com>>> wrote:

markmc did some code that shows how to read the format a few
years
ago here:
http://people.gnome.org/~__markmc/code/merge-dm-snapshot.__c
<http://people.gnome.org/~markmc/code/merge-dm-snapshot.c>
<http://people.gnome.org/%__7Emarkmc/code/merge-dm-__snapshot.c
<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>>


Otherwise look at dm-snap-persistent.c in the kernel tree.

Alasdair


Users can easily exhaust a LiveUSB snapshot overlay by writing
too many changes to the OS filesystem, such as by performing
a yum update.

Would such an 'exhausted' overlay be amenable to some sort of
data recovery or forensics?

If so, how so; if not, why not?


Can you not mount the exhausted dm snapshot? If you mean data
recovery or forensics beyond that, please elaborate.

I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems
before (on actually read-only block devices). Maybe some flags that
I've forgotten get past that, but worst case you can always fake a
writable device with a 2nd overlay/snapshot going to a writable
device, i.e. if you wanted to let fsck repair things.

-dmc


A typical scenario goes like this:

A LiveCD is loaded onto a USB device with a relatively small
persistent overlay and no separate home.img filesystem. The user
proceeds to use the system and save some information. They may
execute a df command and see that there is lots of space
available on the rootfs. Then they decide to execute a yum
update or yum install when there is insufficient space available.
(They do not know about dmsetup status.) Their session will now
not shutdown normally. dmsetup status now shows

live-rw: 0 8388608 snapshot Invalid.

The device fails to complete a subsequent boot attempt. The
startup log shows


You probably know this, but note there is a reset_overlay (or similar)
boot flag which I think causes the overlay/snapshot to be reset, to get
a booting system. So the rest of this is with the use-case in mind of
trying to recover prior contents of the overlay/snapshot in a useful way.




mount: /dev/mapper/live-rw: can't read superblock

e2fsck /dev/dm-0 shows

Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)
e2fsck: Attempt to read block from the filesystem resulted in
short read while trying to open /dev/dm-0
Could this be a zero-length partition?

I recreated the above scenario on Fedora-16-Live-Desktop
installed with an overlay-size-mb of 100. After updating to ~70%
of the overlay, I wrote with
dd if=/dev/zero of=/boot/load bs=1M count=10
3 times to exhaust the overlay.

With the defective device mounted on a good system, and loop
devices set up for ext3fs.img and the overlay file,

dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"
reports
item: 0 8388608 snapshot Invalid


I don't know what that last line of output specifically means as far as
subsequent expectations should go, though presumably others on this list
can elaborate, or worst grep for the message in the code.


Maybe dmsetup is more cooperative with full overlays if they are created
in read-only mode?(wild guess)


I.e. there is certainly the expectation that you won't be able to write
to the device (or at a minimum to any chunk not already in the
overlay(?)). So as I mentioned, the way I'd go about it would be to
recreate it (I'm taking for granted above cmdline is correct), but in
read-only mode, then either just dup that to a new larger device, or add
a second read-write snapshot layer on top (pointing to a looped tmpfile
somewhere). Then try to mount and/or fsck that.


But that presumes that the 'snapshot Invalid' isn't evidence that
something is preventing even read-only access to the full snapshot. For
that I defer to the people more familiar with the relevent code.




e2fsck /dev/dm-0 reports

e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Attempt to read block from filesystem resulted in short
read while trying to open /dev/dm-0
Could this be a zero-length partition?

A hex editor view of the overlay file shows lots of content with
lots of zeros at the end of the file.

My questions are these:

Might there be a way to edit the overlay file to restore its
utility for the information written before the damaging write?

If so, how so; if not, why not?


Were it actually the case that full persistent snapshots become
unusuable (short of 'editing'), that would seem like a bug needing to be
fixed.


-dmc





Suitable references would do as well.

Thanks! --Fred




--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-30-2011, 08:48 PM
Frederick Grose
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On Tue, Nov 29, 2011 at 9:09 PM, Douglas McClendon <dmc.lists@filteredperception.org> wrote:


On 11/29/2011 12:22 AM, Frederick Grose wrote:


On Sun, Nov 27, 2011 at 8:08 PM, Douglas McClendon

<dmc.lists@filteredperception.org

<mailto:dmc.lists@filteredperception.org>> wrote:



* *On 11/27/2011 04:31 PM, Frederick Grose wrote:



* * * *On Sat, Oct 29, 2011 at 4:49 PM, Alasdair G Kergon

* * * *<agk@redhat.com <mailto:agk@redhat.com>

* * * *<mailto:agk@redhat.com <mailto:agk@redhat.com>>> wrote:



* * * * * *markmc did some code that shows how to read the format a few

* * * *years

* * * * * *ago here:

* * * *http://people.gnome.org/~__markmc/code/merge-dm-snapshot.__c

* * * *<http://people.gnome.org/~markmc/code/merge-dm-snapshot.c>

* * * *<http://people.gnome.org/%__7Emarkmc/code/merge-dm-__snapshot.c

* * * *<http://people.gnome.org/%7Emarkmc/code/merge-dm-snapshot.c>>





* * * * * *Otherwise look at dm-snap-persistent.c in the kernel tree.



* * * * * *Alasdair





* * * *Users can easily exhaust a LiveUSB snapshot overlay by writing

* * * *too many changes to the OS filesystem, such as by performing

* * * *a yum update.



* * * *Would such an 'exhausted' overlay be amenable to some sort of

* * * *data recovery or forensics?



* * * *If so, how so; if not, why not?





* *Can you not mount the exhausted dm snapshot? *If you mean data

* *recovery or forensics beyond that, please elaborate.



* *I know I've had issues 'read-only' mounting ext3(2?3?4?) filesystems

* *before (on actually read-only block devices). *Maybe some flags that

* *I've forgotten get past that, but worst case you can always fake a

* *writable device with a 2nd overlay/snapshot going to a writable

* *device, i.e. if you wanted to let fsck repair things.



* *-dmc





A typical scenario goes like this:



A LiveCD is loaded onto a USB device with a relatively small

persistent overlay and no separate home.img filesystem. *The user

proceeds to use the system and save some information. *They may

execute a df command and see that there is lots of space

available on the rootfs. *Then they decide to execute a yum

update or yum install when there is insufficient space available.

(They do not know about dmsetup status.) *Their session will now

not shutdown normally. dmsetup status now shows



live-rw: 0 8388608 snapshot Invalid.



The device fails to complete a subsequent boot attempt. *The

startup log shows




You probably know this, but note there is a reset_overlay (or similar) boot flag which I think causes the overlay/snapshot to be reset, to get a booting system.
Yes, the relevant code for Fedora is here,

http://git.kernel.org/?p=boot/dracut/dracut.git;a=blob;f=modules.d/90dmsquash-live/dmsquash-live-root;hb=HEAD#l101


dd*if=/dev/zero*of=$OVERLAY_LOOPDEV*bs=64k*count=1*2>/dev/null


although it seems to need an extra parameter,*conv=notruncto prevent truncation of the overlay file.


*So the rest of this is with the use-case in mind of trying to recover prior contents of the overlay/snapshot in a useful way.








mount: /dev/mapper/live-rw: can't read superblock



e2fsck /dev/dm-0 shows



Buffer I/O error on device dm-0, logical block 0 (and 1, 2, 3, 0)

e2fsck: Attempt to read block from the filesystem resulted in

short read while trying to open /dev/dm-0

Could this be a zero-length partition?



I recreated the above scenario on Fedora-16-Live-Desktop

installed with an overlay-size-mb of 100. *After updating to ~70%

of the overlay, I wrote with

dd if=/dev/zero of=/boot/load bs=1M count=10

3 times to exhaust the overlay.



With the defective device mounted on a good system, and loop

devices set up for ext3fs.img and the overlay file,



dmsetup create item --table "0 8388608 snapshot 7:1 7:2 P 8"

dmsetup status reports

item: 0 8388608 snapshot Invalid




I don't know what that last line of output specifically means as far as subsequent expectations should go, though presumably others on this list can elaborate, or worst grep for the message in the code.



Maybe dmsetup is more cooperative with full overlays if they are created in read-only mode?(wild guess)

I tested this with --readonly added to the

dmsetup create command, both with and without a-r parameter on the overlay loop device set up command.

Both resulted in the Invalid snapshot report fromdmsetup status.


I.e. there is certainly the expectation that you won't be able to write to the device (or at a minimum to any chunk not already in the overlay(?)). *So as I mentioned, the way I'd go about it would be to recreate it (I'm taking for granted above cmdline is correct), but in read-only mode, then either just dup that to a new larger device, or add a second read-write snapshot layer on top (pointing to a looped tmpfile somewhere). *Then try to mount and/or fsck that.


*Attempting to mount the Invalid snapshot results in,mount: /dev/mapper/item: can't read superblock


A second snapshot set up over the first also fails to mountor e2fsck with the same errors.

*But that presumes that the 'snapshot Invalid' isn't evidence that something is preventing even read-only access to the full snapshot.


snapshot Invalid *does seem to prevent even read-only access.*

For that I defer to the people more familiar with the relevent code.






e2fsck /dev/dm-0 reports



e2fsck 1.41.14 (22-Dec-2010)

e2fsck: Attempt to read block from filesystem resulted in short

read while trying to open /dev/dm-0

Could this be a zero-length partition?



A hex editor view of the overlay file shows lots of content with

lots of zeros at the end of the file.



My questions are these:



Might there be a way to edit the overlay file to restore its

utility for the information written before the damaging write?



If so, how so; if not, why not?




Were it actually the case that full persistent snapshots become unusuable (short of 'editing'), that would seem like a bug needing to be fixed.



-dmc



Suitable references would do as well.



Thanks! * * * --Fred
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-30-2011, 09:03 PM
Alasdair G Kergon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On Wed, Nov 30, 2011 at 04:48:05PM -0500, Frederick Grose wrote:
> snapshot Invalid does seem to prevent even read-only access.

Indeed - once it's invalid, that's the end of it.
If we didn't do that, changes to areas of the origin that hadn't already
changed would start to appear in the 'snapshot' and the snapshot would become a
nonsensical hybrid of the two devices containing no information about which
parts of it got corrupted.

Don't allow your snapshots to run out of space!
If you want to see what you can recover, use that code I mentioned earlier.

Alasdair

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 11-30-2011, 10:34 PM
Douglas McClendon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On 11/30/2011 04:03 PM, Alasdair G Kergon wrote:

On Wed, Nov 30, 2011 at 04:48:05PM -0500, Frederick Grose wrote:

snapshot Invalid does seem to prevent even read-only access.


Indeed - once it's invalid, that's the end of it.
If we didn't do that, changes to areas of the origin that hadn't already
changed would start to appear in the 'snapshot' and the snapshot would become a
nonsensical hybrid of the two devices containing no information about which
parts of it got corrupted.

Don't allow your snapshots to run out of space!


So wouldn't it be much better as far as the user is concerned if, the
instant a snapshot reached capacity, it fell over into a read-only state
(instead of being marked invalid?), such that further corruption beyond
such plug-pulling variety could/would not occur? If so, would it be
straightforward to write such a patch and would it be beneficial enough
to end upstream?


-dmc



If you want to see what you can recover, use that code I mentioned earlier.

Alasdair

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 12-01-2011, 01:12 PM
Alasdair G Kergon
 
Default DM_snapshot_cow filesystem (dmsetup create snapshot)

On Wed, Nov 30, 2011 at 05:34:25PM -0600, Douglas McClendon wrote:
> So wouldn't it be much better as far as the user is concerned if, the
> instant a snapshot reached capacity, it fell over into a read-only state
> (instead of being marked invalid?), such that further corruption beyond

That would mean your *origin* device becoming read-only as well.

It's been discussed before. Nobody has been motivated to write the code,
as there's no need for it on a properly-managed system.

If you care enough about your snapshot contents, then surely you
would be monitoring it to make sure it doesn't fill up - and if you're
going to run out of space and can't expand, you would shut down your system
yourself in a *controlled* way *before* running out of space!

Alasdair

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 05:19 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org