Jigdo - A Professional Letter to Mike McGrath
 

On Thu, 6 Dec 2007 14:28:13 -0700
Jonathan Steffan <jonathansteffan@gmail.com> wrote:

> The amount of storage and bandwidth able to be saved can be
> illustrated by a simple comparison between the efficiency of chopping
> up a 3.4GB iso9660 file system arbitrarily (by a static chunk size)
> and the same file system based on contents (file by file.) For a
> BitTorrent, Fedora's current choice for sharing Spins, the hosted
> data is only valid for a given chunk on a single ISO. This data's
> footprint (equal to the combined chunk sizes of the entire torrent)
> can be used for nothing but this Spin. To be able to host 5 Spins
> composed from similar trees via BitTorrent, we now have a footprint
> of 17GB, not to mention "seeders" have to run BitTorrent software to
> be able to contribute to the swarm. Alternatively, Jigdo can be used
> to reduce the footprint of these 5 Spins to about 4GB. The amount of
> additional data needing to be hosted for each Spin, in addition to
> what data is already pushed to the mirrors, is about 150MB per ISO
> with anaconda and about 200KB for ISOs without the installer bits. To
> help illustrate the efficiency of using Jigdo vs BitTorrent, the
> footprint for 250 Spins is 850GB for BitTorrent and about 40GB for
> Jigdo. Additionally, a reduction in overhead can be achieved by
> removing the need for the BitTorrent tracker and all related network
> traffic without requiring any additional work on the part of mirror
> administrators.

Question. Spins.fp.o is mostly about Live images no? Live images are
essentially a big squasfs image wrapped up in an iso with some bootable
stuff involved? How would jigdo possibly help here?

--
Jesse Keating
Fedora -- All my bits are free, are yours?
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

On Do Dezember 6 2007, Jonathan Steffan wrote:

> The current updates system is getting better each release, but I think
> we should adjust our policies to also have an “updates-archive”
> repository. This repository will include all signed updates that had
> once lived in the updates repo, for the duration of the releases
> life-cycle. I don't expect all mirrors would want to carry this extra

Once the repositories are presto enabled, an updates-archive with all the
delta rpms would not take as much space as a full rpm repository but provide
the same functionality.

Regards,
Till
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

On Thu, 06 Dec 2007 22:53:22 +0100
Till Maas <opensource@till.name> wrote:

> On Do Dezember 6 2007, Jonathan Steffan wrote:
>
> > The current updates system is getting better each release, but I
> > think we should adjust our policies to also have an
> > “updates-archive” repository. This repository will include all
> > signed updates that had once lived in the updates repo, for the
> > duration of the releases life-cycle. I don't expect all mirrors
> > would want to carry this extra
>
> Once the repositories are presto enabled, an updates-archive with all
> the delta rpms would not take as much space as a full rpm repository
> but provide the same functionality.

Yes, this is correct. I left it out of the letter due to the need for a
specific client to deal with the presto data. PyJigdo could do such a
task, if this is the route we want to go.


--
Jonathan Steffan
daMaestro
GPG Fingerprint: 93A2 3E2F DC26 5570 3472 5B16 AD12 6CE7 0D86 AF59

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

On Thu, 6 Dec 2007 16:42:57 -0500
Jesse Keating <jkeating@redhat.com> wrote:.
>
> Question. Spins.fp.o is mostly about Live images no? Live images are
> essentially a big squasfs image wrapped up in an iso with some
> bootable stuff involved? How would jigdo possibly help here?

This is correct. In it's current state, jigdo is only of value for
installer images. I have a goal of getting pyJigdo to be able to deal
with a squashfs, pulling the bits from rpms and then
sticking them into the squash. I have not worked out the technical
details as of yet, but it is planned.

http://pyjigdo.org/


--
Jonathan Steffan
daMaestro
GPG Fingerprint: 93A2 3E2F DC26 5570 3472 5B16 AD12 6CE7 0D86 AF59

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

Jesse Keating wrote:

Question. Spins.fp.o is mostly about Live images no? Live images are
essentially a big squasfs image wrapped up in an iso with some bootable
stuff involved? How would jigdo possibly help here?



As Jigdo needs an expanded tree somewhere (online) of the contents of an
ISO image, so for Live Media, Jigdo isn't a very viable solution.


-Jeroen

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

Till Maas wrote:

On Do Dezember 6 2007, Jonathan Steffan wrote:


The current updates system is getting better each release, but I think
we should adjust our policies to also have an “updates-archive�
repository. This repository will include all signed updates that had
once lived in the updates repo, for the duration of the releases
life-cycle. I don't expect all mirrors would want to carry this extra


Once the repositories are presto enabled, an updates-archive with all the
delta rpms would not take as much space as a full rpm repository but provide
the same functionality.




If in some way a mirror can regenerate a full, original RPM from all
delta RPMs, so that Jigdo can use it as a slice, a combination of the
two would be possible.


Without that ability, all Jigdo recognizes is full RPMs on the ISO image
(slices), against no (zero) matches in the updates-archive/ folder (all
partial slices).


-Jeroen

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

John Poelstra wrote:

Hasn't the door already been opened for this?
http://fedoraproject.org/wiki/ReleaseEngineering/Meetings/2007-nov-19
http://fedoraproject.org/wiki/ReleaseEngineering/Meetings/2007-nov-19#t13:44



The Release Meeting addressed releasing spins via Jigdo, yes.

What had been proposed -and accepted- in the Release Meeting though was
releasing /custom/ spins built of the release tree, via Jigdo. In
particular the Everything Spin comes to mind, composed of the full tree
in releases/$releasever/Everything/$basearch. However, Jonathan's mail
addresses a request to keep signed copies of updates available in
updates/$releasever/$basearch (or somewhere else) for the entire release
cycle, so that:


- Re-Spins and Remixes can be composed at any time during the release
cycle and hosted with the Fedora Project without too large a footprint, and


- Optionally older packages can be used by respinners and remixes, when
needed, for example in case of a yum or rpm update that breaks its
compatibilities with anaconda, or a kernel that just doesn't do the job.




Also, I see an incomplete feature page here:
http://fedoraproject.org/wiki/Features/JigdoRelease




As the owner of the feature; Jigdo hasn't got the integration bits yet
to make full use of Fedora Project infrastructure, as I'm working with
upstream to get some changes done, and pyJigdo (a python alternative
"wrapper" around upstream Jigdo that does integrate it with the existing
Fedora Project in full), hasn't been released stable yet. Then again,
mind that it is an alternative. If people really start using Jigdo it
hits the mirrorlist for every single slice -possibly (or inevitably)
overloading the mirrorlist. With pyJigdo this problem, and another
couple of problems are solved (such as multi-image downloading, error
recovery, scanning more then one local directory for slices, ..., ...).


-Jeroen

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

On Fri, 07 Dec 2007 02:05:15 +0100
Jeroen van Meeuwen <kanarip@kanarip.com> wrote:

> Jesse Keating wrote:
> > Question. Spins.fp.o is mostly about Live images no? Live images
> > are essentially a big squasfs image wrapped up in an iso with some
> > bootable stuff involved? How would jigdo possibly help here?
> >
>
> As Jigdo needs an expanded tree somewhere (online) of the contents of
> an ISO image, so for Live Media, Jigdo isn't a very viable solution.

Well, as I've expressed in
https://www.redhat.com/archives/fedora-infrastructure-list/2007-December/msg00023.html
we could make it so jigdo supports sticking data that has been
published to a tree in some form. To generate the jigdo definition, the
squash would need to be opened and the resulting bits would need to be
hashed. Thus, putting the data back together would also involve
creating a squashfs and sticking data into it. I've not looked into
this outside of just conceptual design. If there are any squashfs gurus
that would like to comment, please do.

--
Jonathan Steffan
daMaestro
GPG Fingerprint: 93A2 3E2F DC26 5570 3472 5B16 AD12 6CE7 0D86 AF59

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

On Fri, 07 Dec 2007 02:10:26 +0100
Jeroen van Meeuwen <kanarip@kanarip.com> wrote:

> Till Maas wrote:
> > On Do Dezember 6 2007, Jonathan Steffan wrote:
> >
> >> The current updates system is getting better each release, but I
> >> think we should adjust our policies to also have an
> >> “updates-archiveâ€_ repository. This repository will include all
> >> signed updates that had once lived in the updates repo, for the
> >> duration of the releases life-cycle. I don't expect all mirrors
> >> would want to carry this extra
> >
> > Once the repositories are presto enabled, an updates-archive with
> > all the delta rpms would not take as much space as a full rpm
> > repository but provide the same functionality.
> >
>
> If in some way a mirror can regenerate a full, original RPM from all
> delta RPMs, so that Jigdo can use it as a slice, a combination of the
> two would be possible.

This would be nice to be able to utilize the mirrors CPU, but most
likely is not practical and we will not be able to accomplished without
mirror admins running special handlers and at that point I would expect
admins would rather waste the disk space vs supporting a home-brew
handler.

> Without that ability, all Jigdo recognizes is full RPMs on the ISO
> image (slices), against no (zero) matches in the updates-archive/
> folder (all partial slices).

Yes, this would require changes to the client software which would
basically obsolete the current jigdo client. We would most likely need
to utilize yum metadata to put all the pieces back together (not a real
problem but it does restrict what platforms we can run pyJigdo on, not
that it runs on anything but fedora at the moment) correctly. Using
presto is also not as bandwidth efficient, but is more disk efficient
(with regards to mirroring an entire tree of archived updates.) I
specially left out using presto in the mix because there would be a
lot of restrictions placed on where the client software could run if we
went this route. I am, however, not in objection to utilizing
existing systems we are developing. We would get the benefit of both
having delta enabled repos for use with yum and a full archive of all
updates released for the duration of a release. Granted multiple deltas
would need to be downloaded to achieve a specific version of a package,
but that also depends on how the deltas are generated.

If anyone knows what the current plan is for generation of deltas,
please chime in. Are we doing deltas against the previous public
release or are we doing deltas against the release package? In both
cases, what is our expected retention policy?

--
Jonathan Steffan
daMaestro
GPG Fingerprint: 93A2 3E2F DC26 5570 3472 5B16 AD12 6CE7 0D86 AF59

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Jigdo - A Professional Letter to Mike McGrath
 

Jonathan Steffan wrote:

The amount of storage and bandwidth able to be saved can be illustrated
by a simple comparison between the efficiency of chopping up a 3.4GB
iso9660 file system arbitrarily (by a static chunk size) and the same
file system based on contents (file by file.) For a BitTorrent,
Fedora's current choice for sharing Spins, the hosted data is only
valid for a given chunk on a single ISO. This data's footprint (equal
to the combined chunk sizes of the entire torrent) can be used for
nothing but this Spin. To be able to host 5 Spins composed from similar
trees via BitTorrent, we now have a footprint of 17GB, not to mention
"seeders" have to run BitTorrent software to be able to contribute to
the swarm. Alternatively, Jigdo can be used to reduce the footprint of
these 5 Spins to about 4GB. The amount of additional data needing to be
hosted for each Spin, in addition to what data is already pushed to the
mirrors, is about 150MB per ISO with anaconda and about 200KB for ISOs
without the installer bits. To help illustrate the efficiency of using
Jigdo vs BitTorrent, the footprint for 250 Spins is 850GB for
BitTorrent and about 40GB for Jigdo. Additionally, a reduction in
overhead can be achieved by removing the need for the BitTorrent
tracker and all related network traffic without requiring any
additional work on the part of mirror administrators.



My concern with jigdo is with how many people use it? It seems silly to
host both torrent and jigdo (as much of this letter points out the
benefits of switching to jigdo, those benefits disappear if we simply
add jigdo to the mix. Most people already have bittorrent. Lets say we
were going to give Jigdo a trial run for Fedora 9 and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?


While google trends is sort of crazy in that I don't know what it says,
it does say something:


http://google.com/trends?q=bittorrent%2C+jigdo

-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list