FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 08-14-2011, 08:07 AM
Ivan Shmakov
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

BTW, I've just announced [1] the availability of the code which
(given some attention and care) may be turned into a Jigdo-like
suite for Ext2+ FS images.

(The casuality of fragmentation on such filesystems greatly
reduces the applicability of the FS-agnostic Jigdo tool.)

Perhaps, this may lessen the burden of the distribution of
Debian virtualization images. (Should it become a part of the
current practice.)

Alternatively, this tool may allow for “smarter” virtualization
images' backups. Or, it may be used to save the bandwidth when
sending a whole virtualization image, e. g., to demonstrate a
hard to reproduce bug.

[1] http://thread.gmane.org/gmane.comp.file-systems.ext4/27269

--
FSF associate member #7257


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86ei0o76wz.fsf@gray.siamics.net">http://lists.debian.org/86ei0o76wz.fsf@gray.siamics.net
 
Old 08-14-2011, 10:15 AM
Adam Borowski
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

On Sun, Aug 14, 2011 at 03:07:24PM +0700, Ivan Shmakov wrote:
> BTW, I've just announced [1] the availability of the code which
> (given some attention and care) may be turned into a Jigdo-like
> suite for Ext2+ FS images.
>
> (The casuality of fragmentation on such filesystems greatly
> reduces the applicability of the FS-agnostic Jigdo tool.)
>
> Perhaps, this may lessen the burden of the distribution of
> Debian virtualization images. (Should it become a part of the
> current practice.)
>
> Alternatively, this tool may allow for “smarter” virtualization
> images' backups. Or, it may be used to save the bandwidth when
> sending a whole virtualization image, e. g., to demonstrate a
> hard to reproduce bug.
>
> [1] http://thread.gmane.org/gmane.comp.file-systems.ext4/27269

It seems to me that if you instead physically reordered blocks, actual data
extents could be made contiguous, allowing use of standard Jigdo.

(Of course, that might be not worth the extra effort...)

--
1KB // Yo momma uses IPv4!


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110814101532.GA6630@angband.pl">http://lists.debian.org/20110814101532.GA6630@angband.pl
 
Old 08-14-2011, 01:20 PM
Ivan Shmakov
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

>>>>> Adam Borowski <kilobyte@angband.pl> writes:
>>>>> On Sun, Aug 14, 2011 at 03:07:24PM +0700, Ivan Shmakov wrote:

>> BTW, I've just announced [1] the availability of the code which
>> (given some attention and care) may be turned into a Jigdo-like
>> suite for Ext2+ FS images.

>> (The casuality of fragmentation on such filesystems greatly reduces
>> the applicability of the FS-agnostic Jigdo tool.)

> It seems to me that if you instead physically reordered blocks,
> actual data extents could be made contiguous, allowing use of
> standard Jigdo.

Thanks for the suggestion.

Unfortunately, the Jigdo's assumption of the continuity of the
blocks comprising a file covers the reassembly phase just as
well.

The blocks of an image may still be reordered to get the desired
effect, but they'd have to be reordered back at the time of
reassembly, thus requiring a separate relation (file) to do so.

Perhaps, the ordering of the blocks on a filesystem can be
altered (consistent with the service data of the filesystem
itself) to remove fragmentation, but I'm not sure on that, and
my limited knowledge of the Ext2FS library doesn't allow me to
devise a way to do that. (But I'd ask about it in linux-ext4@.)

… On the other hand, the “chunk mapping” model I've used in the
current version of the e2dis code is a superset of the model
implemented by Jigdo. Therefore, I don't think it'd be
infeasible to implement Jigdo support in the image download and
reassembly tool I'm planning to develop for e2dis.

(BTW, I've posted a very basic image reassembly tool to
alt.sources about three weeks ago [2]. It's not compatible with
the format used by e2dis, though.)

[2] news:86zkk8xhh4.fsf@gray.siamics.net

> (Of course, that might be not worth the extra effort...)

Jigdo has to “discover” octet sequences on an image with
hashing, which is computationally expensive.

IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
.template file as part of the image generation, which is much
more efficient. I believe that a similar approach could be, and
should be, used for the other filesystems as well.

[…]

>> [1] http://thread.gmane.org/gmane.comp.file-systems.ext4/27269

--
FSF associate member #7257


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86aabc6sf5.fsf@gray.siamics.net">http://lists.debian.org/86aabc6sf5.fsf@gray.siamics.net
 
Old 08-14-2011, 09:08 PM
Steve McIntyre
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

Ivan Shmakov wrote:
>
> Jigdo has to ??discover?? octet sequences on an image with
> hashing, which is computationally expensive.

Yes.

> IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
> .template file as part of the image generation, which is much
> more efficient. I believe that a similar approach could be, and
> should be, used for the other filesystems as well.

That would be absolutely the best way to go, yes. I added the code
into mkisofs and genisoimage, but it has since been abstracted out
into libjte, part of the cdrkit package in Debian.

--
Steve McIntyre, Cambridge, UK. steve@einval.com
"...In the UNIX world, people tend to interpret `non-technical user'
as meaning someone who's only ever written one device driver." -- Daniel Pead


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: E1QshvK-00057x-B7@mail.einval.com">http://lists.debian.org/E1QshvK-00057x-B7@mail.einval.com
 
Old 08-16-2011, 06:06 PM
Ivan Shmakov
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

>>>>> Steve McIntyre <steve@einval.com> writes:
>>>>> Ivan Shmakov wrote:

BTW, the primary Git repository for the project is now located
at Gitorious:

git://gitorious.org/e2dis/e2dis-devel.git
http://gitorious.org/e2dis/e2dis-devel.git
https://gitorious.org/e2dis/e2dis-devel

The most notable improvement that I made recently is the message
digest recording.

[…]

>> IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
>> .template file as part of the image generation, which is much more
>> efficient. I believe that a similar approach could be, and should
>> be, used for the other filesystems as well.

> That would be absolutely the best way to go, yes. I added the code
> into mkisofs and genisoimage, but it has since been abstracted out
> into libjte, part of the cdrkit package in Debian.

I'm yet to take a look at the code, but do I understand it
correctly that Jigdo's .template only allows for a file (as
identified by the SHA-1 digest of its contents) to be associated
with a contiguous part of the destination image?

Such an approach is hardly applicable to Ext2+ FS images.
Consider, e. g.:

$ /sbin/debugfs +b21e37e0-c722-11e0-9094-001966aaa0b6.ext2
debugfs 1.41.12 (17-May-2010)
debugfs: stat <12>
Inode: 12 Type: regular Mode: 0644 Flags: 0x0
Generation: 3178327445 Version: 0x00000000
User: 0 Group: 0 Size: 16384
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 34
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4e48e9c1 -- Mon Aug 15 09:41:21 2011
atime: 0x4e48e9c1 -- Mon Aug 15 09:41:21 2011
mtime: 0x4e48e9c1 -- Mon Aug 15 09:41:21 2011
BLOCKS:
(0-11):21-32, (IND):33, (12-15):34-37
TOTAL: 17

debugfs:

There, the file's contents in two parts: one occupies image's
blocks 21 to 32, inclusive, and the other one 34 to 37,
inclusive.

It was my understanding that Jigdo's .template cannot associate
one SHA-1 with such a non-contiguous chunk.

--
FSF associate member #7257


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86wred1bac.fsf@gray.siamics.net">http://lists.debian.org/86wred1bac.fsf@gray.siamics.net
 
Old 09-02-2011, 01:56 PM
Steve McIntyre
 
Default e2dis: a Jigdo-like tool for Ext2+ FS images

Ivan Shmakov wrote:
>
> It was my understanding that Jigdo's .template cannot associate
> one SHA-1 with such a non-contiguous chunk.

Correct. That could be added as a feature, maybe - we could add
extent-mapping. I'm also thinking of adding code to say "this file
from within this .deb", so we could maybe support jigdo for live
images too...

--
Steve McIntyre, Cambridge, UK. steve@einval.com
Mature Sporty Personal
More Innovation More Adult
A Man in Dandism
Powered Midship Specialty


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: E1QzUDy-0006ZL-T3@mail.einval.com">http://lists.debian.org/E1QzUDy-0006ZL-T3@mail.einval.com
 

Thread Tools




All times are GMT. The time now is 01:30 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org