BTW, I've just announced [1] the availability of the code which
(given some attention and care) may be turned into a Jigdo-like
suite for Ext2+ FS images.
(The casuality of fragmentation on such filesystems greatly
reduces the applicability of the FS-agnostic Jigdo tool.)
Perhaps, this may lessen the burden of the distribution of
Debian virtualization images. (Should it become a part of the
current practice.)
Alternatively, this tool may allow for “smarter” virtualization
images' backups. Or, it may be used to save the bandwidth when
sending a whole virtualization image, e. g., to demonstrate a
hard to reproduce bug.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86ei0o76wz.fsf@gray.siamics.net">http://lists.debian.org/86ei0o76wz.fsf@gray.siamics.net
08-14-2011, 10:15 AM
Adam Borowski
e2dis: a Jigdo-like tool for Ext2+ FS images
On Sun, Aug 14, 2011 at 03:07:24PM +0700, Ivan Shmakov wrote:
> BTW, I've just announced [1] the availability of the code which
> (given some attention and care) may be turned into a Jigdo-like
> suite for Ext2+ FS images.
>
> (The casuality of fragmentation on such filesystems greatly
> reduces the applicability of the FS-agnostic Jigdo tool.)
>
> Perhaps, this may lessen the burden of the distribution of
> Debian virtualization images. (Should it become a part of the
> current practice.)
>
> Alternatively, this tool may allow for “smarter” virtualization
> images' backups. Or, it may be used to save the bandwidth when
> sending a whole virtualization image, e. g., to demonstrate a
> hard to reproduce bug.
>
> [1] http://thread.gmane.org/gmane.comp.file-systems.ext4/27269
It seems to me that if you instead physically reordered blocks, actual data
extents could be made contiguous, allowing use of standard Jigdo.
(Of course, that might be not worth the extra effort...)
--
1KB // Yo momma uses IPv4!
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110814101532.GA6630@angband.pl">http://lists.debian.org/20110814101532.GA6630@angband.pl
08-14-2011, 01:20 PM
Ivan Shmakov
e2dis: a Jigdo-like tool for Ext2+ FS images
>>>>> Adam Borowski <kilobyte@angband.pl> writes:
>>>>> On Sun, Aug 14, 2011 at 03:07:24PM +0700, Ivan Shmakov wrote:
>> BTW, I've just announced [1] the availability of the code which
>> (given some attention and care) may be turned into a Jigdo-like
>> suite for Ext2+ FS images.
>> (The casuality of fragmentation on such filesystems greatly reduces
>> the applicability of the FS-agnostic Jigdo tool.)
> It seems to me that if you instead physically reordered blocks,
> actual data extents could be made contiguous, allowing use of
> standard Jigdo.
Thanks for the suggestion.
Unfortunately, the Jigdo's assumption of the continuity of the
blocks comprising a file covers the reassembly phase just as
well.
The blocks of an image may still be reordered to get the desired
effect, but they'd have to be reordered back at the time of
reassembly, thus requiring a separate relation (file) to do so.
Perhaps, the ordering of the blocks on a filesystem can be
altered (consistent with the service data of the filesystem
itself) to remove fragmentation, but I'm not sure on that, and
my limited knowledge of the Ext2FS library doesn't allow me to
devise a way to do that. (But I'd ask about it in linux-ext4@.)
… On the other hand, the “chunk mapping” model I've used in the
current version of the e2dis code is a superset of the model
implemented by Jigdo. Therefore, I don't think it'd be
infeasible to implement Jigdo support in the image download and
reassembly tool I'm planning to develop for e2dis.
(BTW, I've posted a very basic image reassembly tool to
alt.sources about three weeks ago [2]. It's not compatible with
the format used by e2dis, though.)
[2] news:86zkk8xhh4.fsf@gray.siamics.net
> (Of course, that might be not worth the extra effort...)
Jigdo has to “discover” octet sequences on an image with
hashing, which is computationally expensive.
IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
.template file as part of the image generation, which is much
more efficient. I believe that a similar approach could be, and
should be, used for the other filesystems as well.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86aabc6sf5.fsf@gray.siamics.net">http://lists.debian.org/86aabc6sf5.fsf@gray.siamics.net
08-14-2011, 09:08 PM
Steve McIntyre
e2dis: a Jigdo-like tool for Ext2+ FS images
Ivan Shmakov wrote:
>
> Jigdo has to ??discover?? octet sequences on an image with
> hashing, which is computationally expensive.
Yes.
> IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
> .template file as part of the image generation, which is much
> more efficient. I believe that a similar approach could be, and
> should be, used for the other filesystems as well.
That would be absolutely the best way to go, yes. I added the code
into mkisofs and genisoimage, but it has since been abstracted out
into libjte, part of the cdrkit package in Debian.
--
Steve McIntyre, Cambridge, UK. steve@einval.com
"...In the UNIX world, people tend to interpret `non-technical user'
as meaning someone who's only ever written one device driver." -- Daniel Pead
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: E1QshvK-00057x-B7@mail.einval.com">http://lists.debian.org/E1QshvK-00057x-B7@mail.einval.com
08-16-2011, 06:06 PM
Ivan Shmakov
e2dis: a Jigdo-like tool for Ext2+ FS images
>>>>> Steve McIntyre <steve@einval.com> writes:
>>>>> Ivan Shmakov wrote:
BTW, the primary Git repository for the project is now located
at Gitorious:
The most notable improvement that I made recently is the message
digest recording.
[…]
>> IIRC, Debian's genisoimage(1) was altered to prepare a Jigdo's
>> .template file as part of the image generation, which is much more
>> efficient. I believe that a similar approach could be, and should
>> be, used for the other filesystems as well.
> That would be absolutely the best way to go, yes. I added the code
> into mkisofs and genisoimage, but it has since been abstracted out
> into libjte, part of the cdrkit package in Debian.
I'm yet to take a look at the code, but do I understand it
correctly that Jigdo's .template only allows for a file (as
identified by the SHA-1 digest of its contents) to be associated
with a contiguous part of the destination image?
Such an approach is hardly applicable to Ext2+ FS images.
Consider, e. g.:
There, the file's contents in two parts: one occupies image's
blocks 21 to 32, inclusive, and the other one 34 to 37,
inclusive.
It was my understanding that Jigdo's .template cannot associate
one SHA-1 with such a non-contiguous chunk.
--
FSF associate member #7257
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 86wred1bac.fsf@gray.siamics.net">http://lists.debian.org/86wred1bac.fsf@gray.siamics.net
09-02-2011, 01:56 PM
Steve McIntyre
e2dis: a Jigdo-like tool for Ext2+ FS images
Ivan Shmakov wrote:
>
> It was my understanding that Jigdo's .template cannot associate
> one SHA-1 with such a non-contiguous chunk.
Correct. That could be added as a feature, maybe - we could add
extent-mapping. I'm also thinking of adding code to say "this file
from within this .deb", so we could maybe support jigdo for live
images too...
--
Steve McIntyre, Cambridge, UK. steve@einval.com
Mature Sporty Personal
More Innovation More Adult
A Man in Dandism
Powered Midship Specialty
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: E1QzUDy-0006ZL-T3@mail.einval.com">http://lists.debian.org/E1QzUDy-0006ZL-T3@mail.einval.com
e2dis: a Jigdo-like tool for Ext2+ FS images
