FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 01-05-2012, 07:04 PM
Sander Eikelenboom
 
Default can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd

Thursday, January 5, 2012, 7:15:35 PM, you wrote:

> On Thu, Jan 05, 2012 at 05:14:28PM +0100, Sander Eikelenboom wrote:
>>
>> OK spoke too soon, i have been able to trigger it again:
>> - copying files from LV to the same LV without the snapshot went OK
>> - copying from the RO snapshot of a LV to the same LV gave the error while copying the file again:

> OK. Originally, you said you did this:

> 1) fsck -v -p -f the filesystem
> 2) mount the filesystem
> 3) Try to copy a file
> 4) filesystem will be mounted RO on error (see below)
> 5) fsck again, journal will be recovered, no other errors
> 6) start at 1)

> Was this with with a read-only snapshot always being in existence
> through all of these five steps? When was the RO snapshot created?

> If a RO snapshot has to be there in order for this to happen, then
> this is almost certainly a device-mapper regression. (dm-devel folks,
> this is a problem which apparently occurred when the user went from
> v3.1.5 to v3.2, so this looks likes 3.2 regression.)

> - Ted


Well it seems to consist of 2 issues with a kernel booted with a 3.2.0 kernel:

1) - It only seems to trigger with a snapshot of the LV present
- Just tested if the snapshot being mounted RO did really matter, it doesn't.
- It can also be triggerd if mounted RW
- It can also be triggered when the snapshot is not mounted at all (by just copying some files on the filesystem itself)

So that seems a device mapper issue

2) BUT:
after the error triggerd by 1:
- After removing the snapshot with lvremove,
- umounting the filesystem on the LV
- fsckíng the filesystem without errors (apart from the journal recovery)
- rebooting the machine again with 3.2.0 kernel
- mounting the filesystem on the LV
- removing the partially copied files
- trying to copy files from the filesystem on the LV to the same filesystem, without a snapshot of the LV present
- it fails with the exact same error mounting the filesystem RO.

then
- umounting the filesystem on the LV
- fsckíng the filesystem without errors (apart from the journal recovery)
- rebooting the machine with a 3.1.5 kernel
- mounting the filesystem on the LV
- removing the partially copied files
- trying to copy files from the filesystem on the LV to the same filesystem, without a snapshot of the LV present
- no problems files copied ok

then
- rebooting into 3.2.0 again
- mounting the filesystem on the LV
- removing the completly copied files
- trying to copy files from the filesystem on the LV to the same filesystem, without a snapshot of the LV present
- no problems files copied ok


SO
- it keeps on failing on 3.2.0, even when the snapshot is gone and the system is rebooted, after 3.1.5 is booted once everything seems to be OK again .... even under 3.2.0
- that seems more like a filesystem thing ?


I doubled checked and performed all these steps again.

--
Sander








>>
>> [ 2357.655783] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1861, 32254 clusters in bitmap, 32258 in gd
>> [ 2357.656056] Aborting journal on device dm-2-8.
>> [ 2357.718473] EXT4-fs (dm-2): Remounting filesystem read-only
>> [ 2357.736680] EXT4-fs error (device dm-2) in ext4_da_write_end:2532: IO failure
>> [ 2357.738328] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 7615 pages, ino 4079617; err -30
>> [ 2716.125010] EXT4-fs error (device dm-2): ext4_put_super:818: Couldn't clean up the journal
>>
>>
>> Attached are 4x output from dumpe2fs
>> - dumpe2fs-xen_images-3.2.0 Made just after boot
>> - dumpe2fs-xen_images-3.2.0-afterfsck Made after doing a fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror Made after the error occured on the mounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror-afterfsck Made after the error occured, and after a subsequent fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.1.5 Made after booting into 3.1.5 after all of the above
>>
>> Oh yes also did a badblock scan to rule that out, and it seems the numbers stay the same.
>> e2fsck 1.41.12 (17-May-2010) (from debian squeeze)
>>
>> --
>> Sander
>>
>>
>>
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> This is a forwarded message
>> >> From: Sander Eikelenboom <linux@eikelenboom.it>
>> >> To: "Theodore Ts'o" <tytso@mit.edu>
>> >> Date: Thursday, January 5, 2012, 11:37:59 AM
>> >> Subject: can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >>
>> >> ===8<==============Original message text===============
>> >>
>> >> I'm having some troubles with a ext4 filesystem on LVM, it seems bricked and fsck doesn't seem to find and correct the problem.
>> >>
>> >> Steps:
>> >> 1) fsck -v -p -f the filesystem
>> >> 2) mount the filesystem
>> >> 3) Try to copy a file
>> >> 4) filesystem will be mounted RO on error (see below)
>> >> 5) fsck again, journal will be recovered, no other errors
>> >> 6) start at 1)
>> >>
>> >>
>> >> I think the way i bricked it is:
>> >> - make a lvm snapshot from that lvm logical disk
>> >> - mount that lvm snapshot as RO
>> >> - try to copy a file from that mounted RO snapshot to a diffrent dir on the lvm logical disk the snapshot is from.
>> >> - it fails and i can't recover (see above)
>> >>
>> >>
>> >> Is there a way to recover from this ?
>> >>
>> >>
>> >>
>> >> [ 220.748928] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >> [ 220.749415] Aborting journal on device dm-2-8.
>> >> [ 220.771633] EXT4-fs error (device dm-2): ext4_journal_start_sb:327: Detected aborted journal
>> >> [ 220.772593] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.792455] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.805118] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 9680 pages, ino 4079617; err -30
>> >> serveerstertje:/mnt/xen_images/domains/production# cd /
>> >> serveerstertje:/# umount /mnt/xen_images/
>> >> serveerstertje:/# fsck -f -v -p /dev/serveerstertje/xen_images
>> >> fsck from util-linux-ng 2.17.2
>> >> /dev/mapper/serveerstertje-xen_images: recovering journal
>> >>
>> >> 277 inodes used (0.00%)
>> >> 5 non-contiguous files (1.8%)
>> >> 0 non-contiguous directories (0.0%)
>> >> # of inodes with ind/dind/tind blocks: 41/41/3
>> >> Extent depth histogram: 69/28/2
>> >> 51890920 blocks used (79.18%)
>> >> 0 bad blocks
>> >> 41 large files
>> >>
>> >> 199 regular files
>> >> 53 directories
>> >> 0 character device files
>> >> 0 block device files
>> >> 0 fifos
>> >> 0 links
>> >> 16 symbolic links (16 fast symbolic links)
>> >> 0 sockets
>> >> --------
>> >> 268 files
>> >> serveerstertje:/#
>> >>
>> >>
>> >>
>> >>
>> >> System:
>> >> - Kernel 3.2.0
>> >> - Debian Squeeze with:
>> >> ii e2fslibs 1.41.12-4stable1 ext2/ext3/ext4 file system libraries
>> >> ii e2fsprogs 1.41.12-4stable1 ext2/ext3/ext4 file system utilities
>> >>
>> >> ===8<===========End of original message text===========
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> Sander mailto:linux@eikelenboom.it<Message01.eml>
>>
>>
>>
>>
>> --
>> Best regards,
>> Sander mailto:linux@eikelenboom.it









--
Best regards,
Sander mailto:linux@eikelenboom.it


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-05-2012, 07:45 PM
Sander Eikelenboom
 
Default can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd

Thursday, January 5, 2012, 7:15:35 PM, you wrote:

> On Thu, Jan 05, 2012 at 05:14:28PM +0100, Sander Eikelenboom wrote:
>>
>> OK spoke too soon, i have been able to trigger it again:
>> - copying files from LV to the same LV without the snapshot went OK
>> - copying from the RO snapshot of a LV to the same LV gave the error while copying the file again:

> OK. Originally, you said you did this:

> 1) fsck -v -p -f the filesystem
> 2) mount the filesystem
> 3) Try to copy a file
> 4) filesystem will be mounted RO on error (see below)
> 5) fsck again, journal will be recovered, no other errors
> 6) start at 1)

> Was this with with a read-only snapshot always being in existence
> through all of these five steps? When was the RO snapshot created?

> If a RO snapshot has to be there in order for this to happen, then
> this is almost certainly a device-mapper regression. (dm-devel folks,
> this is a problem which apparently occurred when the user went from
> v3.1.5 to v3.2, so this looks likes 3.2 regression.)

> - Ted


Also found some old info that might be related, http://answers.softpicks.net/answers/topic/2-6-28-ext4-xen-and-lvm-volume-becomes-ro-after-snapshot-1610734-1.htm
I'm also running under xen (host only, no guests started), will try baremetal as well to see if that plays a role.

--
Sander


>>
>> [ 2357.655783] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1861, 32254 clusters in bitmap, 32258 in gd
>> [ 2357.656056] Aborting journal on device dm-2-8.
>> [ 2357.718473] EXT4-fs (dm-2): Remounting filesystem read-only
>> [ 2357.736680] EXT4-fs error (device dm-2) in ext4_da_write_end:2532: IO failure
>> [ 2357.738328] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 7615 pages, ino 4079617; err -30
>> [ 2716.125010] EXT4-fs error (device dm-2): ext4_put_super:818: Couldn't clean up the journal
>>
>>
>> Attached are 4x output from dumpe2fs
>> - dumpe2fs-xen_images-3.2.0 Made just after boot
>> - dumpe2fs-xen_images-3.2.0-afterfsck Made after doing a fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror Made after the error occured on the mounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror-afterfsck Made after the error occured, and after a subsequent fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.1.5 Made after booting into 3.1.5 after all of the above
>>
>> Oh yes also did a badblock scan to rule that out, and it seems the numbers stay the same.
>> e2fsck 1.41.12 (17-May-2010) (from debian squeeze)
>>
>> --
>> Sander
>>
>>
>>
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> This is a forwarded message
>> >> From: Sander Eikelenboom <linux@eikelenboom.it>
>> >> To: "Theodore Ts'o" <tytso@mit.edu>
>> >> Date: Thursday, January 5, 2012, 11:37:59 AM
>> >> Subject: can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >>
>> >> ===8<==============Original message text===============
>> >>
>> >> I'm having some troubles with a ext4 filesystem on LVM, it seems bricked and fsck doesn't seem to find and correct the problem.
>> >>
>> >> Steps:
>> >> 1) fsck -v -p -f the filesystem
>> >> 2) mount the filesystem
>> >> 3) Try to copy a file
>> >> 4) filesystem will be mounted RO on error (see below)
>> >> 5) fsck again, journal will be recovered, no other errors
>> >> 6) start at 1)
>> >>
>> >>
>> >> I think the way i bricked it is:
>> >> - make a lvm snapshot from that lvm logical disk
>> >> - mount that lvm snapshot as RO
>> >> - try to copy a file from that mounted RO snapshot to a diffrent dir on the lvm logical disk the snapshot is from.
>> >> - it fails and i can't recover (see above)
>> >>
>> >>
>> >> Is there a way to recover from this ?
>> >>
>> >>
>> >>
>> >> [ 220.748928] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >> [ 220.749415] Aborting journal on device dm-2-8.
>> >> [ 220.771633] EXT4-fs error (device dm-2): ext4_journal_start_sb:327: Detected aborted journal
>> >> [ 220.772593] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.792455] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.805118] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 9680 pages, ino 4079617; err -30
>> >> serveerstertje:/mnt/xen_images/domains/production# cd /
>> >> serveerstertje:/# umount /mnt/xen_images/
>> >> serveerstertje:/# fsck -f -v -p /dev/serveerstertje/xen_images
>> >> fsck from util-linux-ng 2.17.2
>> >> /dev/mapper/serveerstertje-xen_images: recovering journal
>> >>
>> >> 277 inodes used (0.00%)
>> >> 5 non-contiguous files (1.8%)
>> >> 0 non-contiguous directories (0.0%)
>> >> # of inodes with ind/dind/tind blocks: 41/41/3
>> >> Extent depth histogram: 69/28/2
>> >> 51890920 blocks used (79.18%)
>> >> 0 bad blocks
>> >> 41 large files
>> >>
>> >> 199 regular files
>> >> 53 directories
>> >> 0 character device files
>> >> 0 block device files
>> >> 0 fifos
>> >> 0 links
>> >> 16 symbolic links (16 fast symbolic links)
>> >> 0 sockets
>> >> --------
>> >> 268 files
>> >> serveerstertje:/#
>> >>
>> >>
>> >>
>> >>
>> >> System:
>> >> - Kernel 3.2.0
>> >> - Debian Squeeze with:
>> >> ii e2fslibs 1.41.12-4stable1 ext2/ext3/ext4 file system libraries
>> >> ii e2fsprogs 1.41.12-4stable1 ext2/ext3/ext4 file system utilities
>> >>
>> >> ===8<===========End of original message text===========
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> Sander mailto:linux@eikelenboom.it<Message01.eml>
>>
>>
>>
>>
>> --
>> Best regards,
>> Sander mailto:linux@eikelenboom.it









--
Best regards,
Sander mailto:linux@eikelenboom.it

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-05-2012, 08:31 PM
Sander Eikelenboom
 
Default can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd

Hello Ted,

Thursday, January 5, 2012, 7:15:35 PM, you wrote:

> On Thu, Jan 05, 2012 at 05:14:28PM +0100, Sander Eikelenboom wrote:
>>
>> OK spoke too soon, i have been able to trigger it again:
>> - copying files from LV to the same LV without the snapshot went OK
>> - copying from the RO snapshot of a LV to the same LV gave the error while copying the file again:

> OK. Originally, you said you did this:

> 1) fsck -v -p -f the filesystem
> 2) mount the filesystem
> 3) Try to copy a file
> 4) filesystem will be mounted RO on error (see below)
> 5) fsck again, journal will be recovered, no other errors
> 6) start at 1)

> Was this with with a read-only snapshot always being in existence
> through all of these five steps? When was the RO snapshot created?

> If a RO snapshot has to be there in order for this to happen, then
> this is almost certainly a device-mapper regression. (dm-devel folks,
> this is a problem which apparently occurred when the user went from
> v3.1.5 to v3.2, so this looks likes 3.2 regression.)

> - Ted


OK Xen is out of the equation, it also happens on baremetal.
Last time under both Xen and baremetal i got a slightly different error (different numbers (group)

[ 823.782633] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1865, 32254 clusters in bitmap, 32258 in gd
[ 823.788129] Aborting journal on device dm-2-8.
[ 823.852443] EXT4-fs (dm-2): Remounting filesystem read-only
[ 823.857956] EXT4-fs error (device dm-2) in ext4_da_write_end:2532: IO failure
[ 823.858646] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 12288 pages, ino 4079617; err -30



>>
>> [ 2357.655783] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1861, 32254 clusters in bitmap, 32258 in gd
>> [ 2357.656056] Aborting journal on device dm-2-8.
>> [ 2357.718473] EXT4-fs (dm-2): Remounting filesystem read-only
>> [ 2357.736680] EXT4-fs error (device dm-2) in ext4_da_write_end:2532: IO failure
>> [ 2357.738328] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 7615 pages, ino 4079617; err -30
>> [ 2716.125010] EXT4-fs error (device dm-2): ext4_put_super:818: Couldn't clean up the journal
>>
>>
>> Attached are 4x output from dumpe2fs
>> - dumpe2fs-xen_images-3.2.0 Made just after boot
>> - dumpe2fs-xen_images-3.2.0-afterfsck Made after doing a fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror Made after the error occured on the mounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror-afterfsck Made after the error occured, and after a subsequent fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.1.5 Made after booting into 3.1.5 after all of the above
>>
>> Oh yes also did a badblock scan to rule that out, and it seems the numbers stay the same.
>> e2fsck 1.41.12 (17-May-2010) (from debian squeeze)
>>
>> --
>> Sander
>>
>>
>>
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> This is a forwarded message
>> >> From: Sander Eikelenboom <linux@eikelenboom.it>
>> >> To: "Theodore Ts'o" <tytso@mit.edu>
>> >> Date: Thursday, January 5, 2012, 11:37:59 AM
>> >> Subject: can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >>
>> >> ===8<==============Original message text===============
>> >>
>> >> I'm having some troubles with a ext4 filesystem on LVM, it seems bricked and fsck doesn't seem to find and correct the problem.
>> >>
>> >> Steps:
>> >> 1) fsck -v -p -f the filesystem
>> >> 2) mount the filesystem
>> >> 3) Try to copy a file
>> >> 4) filesystem will be mounted RO on error (see below)
>> >> 5) fsck again, journal will be recovered, no other errors
>> >> 6) start at 1)
>> >>
>> >>
>> >> I think the way i bricked it is:
>> >> - make a lvm snapshot from that lvm logical disk
>> >> - mount that lvm snapshot as RO
>> >> - try to copy a file from that mounted RO snapshot to a diffrent dir on the lvm logical disk the snapshot is from.
>> >> - it fails and i can't recover (see above)
>> >>
>> >>
>> >> Is there a way to recover from this ?
>> >>
>> >>
>> >>
>> >> [ 220.748928] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >> [ 220.749415] Aborting journal on device dm-2-8.
>> >> [ 220.771633] EXT4-fs error (device dm-2): ext4_journal_start_sb:327: Detected aborted journal
>> >> [ 220.772593] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.792455] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.805118] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 9680 pages, ino 4079617; err -30
>> >> serveerstertje:/mnt/xen_images/domains/production# cd /
>> >> serveerstertje:/# umount /mnt/xen_images/
>> >> serveerstertje:/# fsck -f -v -p /dev/serveerstertje/xen_images
>> >> fsck from util-linux-ng 2.17.2
>> >> /dev/mapper/serveerstertje-xen_images: recovering journal
>> >>
>> >> 277 inodes used (0.00%)
>> >> 5 non-contiguous files (1.8%)
>> >> 0 non-contiguous directories (0.0%)
>> >> # of inodes with ind/dind/tind blocks: 41/41/3
>> >> Extent depth histogram: 69/28/2
>> >> 51890920 blocks used (79.18%)
>> >> 0 bad blocks
>> >> 41 large files
>> >>
>> >> 199 regular files
>> >> 53 directories
>> >> 0 character device files
>> >> 0 block device files
>> >> 0 fifos
>> >> 0 links
>> >> 16 symbolic links (16 fast symbolic links)
>> >> 0 sockets
>> >> --------
>> >> 268 files
>> >> serveerstertje:/#
>> >>
>> >>
>> >>
>> >>
>> >> System:
>> >> - Kernel 3.2.0
>> >> - Debian Squeeze with:
>> >> ii e2fslibs 1.41.12-4stable1 ext2/ext3/ext4 file system libraries
>> >> ii e2fsprogs 1.41.12-4stable1 ext2/ext3/ext4 file system utilities
>> >>
>> >> ===8<===========End of original message text===========
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> Sander mailto:linux@eikelenboom.it<Message01.eml>
>>
>>
>>
>>
>> --
>> Best regards,
>> Sander mailto:linux@eikelenboom.it









--
Best regards,
Sander mailto:linux@eikelenboom.it

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-05-2012, 09:43 PM
Sander Eikelenboom
 
Default can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd

Hello Ted,

Thursday, January 5, 2012, 7:15:35 PM, you wrote:

> On Thu, Jan 05, 2012 at 05:14:28PM +0100, Sander Eikelenboom wrote:
>>
>> OK spoke too soon, i have been able to trigger it again:
>> - copying files from LV to the same LV without the snapshot went OK
>> - copying from the RO snapshot of a LV to the same LV gave the error while copying the file again:

> OK. Originally, you said you did this:

> 1) fsck -v -p -f the filesystem
> 2) mount the filesystem
> 3) Try to copy a file
> 4) filesystem will be mounted RO on error (see below)
> 5) fsck again, journal will be recovered, no other errors
> 6) start at 1)

> Was this with with a read-only snapshot always being in existence
> through all of these five steps? When was the RO snapshot created?

> If a RO snapshot has to be there in order for this to happen, then
> this is almost certainly a device-mapper regression. (dm-devel folks,
> this is a problem which apparently occurred when the user went from
> v3.1.5 to v3.2, so this looks likes 3.2 regression.)

> - Ted

Tried to bisect, but every kernel in between seems to have some drivers for devices f*cked up so it doesn't even boot.
That was a quite frustrating and disappointing experience.
So it's back to 3.1.5 and continue with i was actually trying to do, and try later if it's still reproducible with another disk layout.

Thx for your effort so far.

--
Sander

>>
>> [ 2357.655783] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1861, 32254 clusters in bitmap, 32258 in gd
>> [ 2357.656056] Aborting journal on device dm-2-8.
>> [ 2357.718473] EXT4-fs (dm-2): Remounting filesystem read-only
>> [ 2357.736680] EXT4-fs error (device dm-2) in ext4_da_write_end:2532: IO failure
>> [ 2357.738328] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 7615 pages, ino 4079617; err -30
>> [ 2716.125010] EXT4-fs error (device dm-2): ext4_put_super:818: Couldn't clean up the journal
>>
>>
>> Attached are 4x output from dumpe2fs
>> - dumpe2fs-xen_images-3.2.0 Made just after boot
>> - dumpe2fs-xen_images-3.2.0-afterfsck Made after doing a fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror Made after the error occured on the mounted LV
>> - dumpe2fs-xen_images-3.2.0-aftererror-afterfsck Made after the error occured, and after a subsequent fsck -v -p -f on the unmounted LV
>> - dumpe2fs-xen_images-3.1.5 Made after booting into 3.1.5 after all of the above
>>
>> Oh yes also did a badblock scan to rule that out, and it seems the numbers stay the same.
>> e2fsck 1.41.12 (17-May-2010) (from debian squeeze)
>>
>> --
>> Sander
>>
>>
>>
>> >>
>> >> --
>> >> Sander
>> >>
>> >>
>> >> This is a forwarded message
>> >> From: Sander Eikelenboom <linux@eikelenboom.it>
>> >> To: "Theodore Ts'o" <tytso@mit.edu>
>> >> Date: Thursday, January 5, 2012, 11:37:59 AM
>> >> Subject: can't recover ext4 on lvm from ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >>
>> >> ===8<==============Original message text===============
>> >>
>> >> I'm having some troubles with a ext4 filesystem on LVM, it seems bricked and fsck doesn't seem to find and correct the problem.
>> >>
>> >> Steps:
>> >> 1) fsck -v -p -f the filesystem
>> >> 2) mount the filesystem
>> >> 3) Try to copy a file
>> >> 4) filesystem will be mounted RO on error (see below)
>> >> 5) fsck again, journal will be recovered, no other errors
>> >> 6) start at 1)
>> >>
>> >>
>> >> I think the way i bricked it is:
>> >> - make a lvm snapshot from that lvm logical disk
>> >> - mount that lvm snapshot as RO
>> >> - try to copy a file from that mounted RO snapshot to a diffrent dir on the lvm logical disk the snapshot is from.
>> >> - it fails and i can't recover (see above)
>> >>
>> >>
>> >> Is there a way to recover from this ?
>> >>
>> >>
>> >>
>> >> [ 220.748928] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:739: group 1687, 32254 clusters in bitmap, 32258 in gd
>> >> [ 220.749415] Aborting journal on device dm-2-8.
>> >> [ 220.771633] EXT4-fs error (device dm-2): ext4_journal_start_sb:327: Detected aborted journal
>> >> [ 220.772593] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.792455] EXT4-fs (dm-2): Remounting filesystem read-only
>> >> [ 220.805118] EXT4-fs (dm-2): ext4_da_writepages: jbd2_start: 9680 pages, ino 4079617; err -30
>> >> serveerstertje:/mnt/xen_images/domains/production# cd /
>> >> serveerstertje:/# umount /mnt/xen_images/
>> >> serveerstertje:/# fsck -f -v -p /dev/serveerstertje/xen_images
>> >> fsck from util-linux-ng 2.17.2
>> >> /dev/mapper/serveerstertje-xen_images: recovering journal
>> >>
>> >> 277 inodes used (0.00%)
>> >> 5 non-contiguous files (1.8%)
>> >> 0 non-contiguous directories (0.0%)
>> >> # of inodes with ind/dind/tind blocks: 41/41/3
>> >> Extent depth histogram: 69/28/2
>> >> 51890920 blocks used (79.18%)
>> >> 0 bad blocks
>> >> 41 large files
>> >>
>> >> 199 regular files
>> >> 53 directories
>> >> 0 character device files
>> >> 0 block device files
>> >> 0 fifos
>> >> 0 links
>> >> 16 symbolic links (16 fast symbolic links)
>> >> 0 sockets
>> >> --------
>> >> 268 files
>> >> serveerstertje:/#
>> >>
>> >>
>> >>
>> >>
>> >> System:
>> >> - Kernel 3.2.0
>> >> - Debian Squeeze with:
>> >> ii e2fslibs 1.41.12-4stable1 ext2/ext3/ext4 file system libraries
>> >> ii e2fsprogs 1.41.12-4stable1 ext2/ext3/ext4 file system utilities
>> >>
>> >> ===8<===========End of original message text===========
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> Sander mailto:linux@eikelenboom.it<Message01.eml>
>>
>>
>>
>>
>> --
>> Best regards,
>> Sander mailto:linux@eikelenboom.it









--
Best regards,
Sander mailto:linux@eikelenboom.it

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 12:07 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org