FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Infrastructure

 
 
LinkBack Thread Tools
 
Old 12-07-2007, 02:14 PM
Jeroen van Meeuwen
 
Default Jigdo - A Professional Letter to Mike McGrath

Mike McGrath wrote:

Jonathan Steffan wrote:

The amount of storage and bandwidth able to be saved can be illustrated
by a simple comparison between the efficiency of chopping up a 3.4GB
iso9660 file system arbitrarily (by a static chunk size) and the same
file system based on contents (file by file.) For a BitTorrent,
Fedora's current choice for sharing Spins, the hosted data is only
valid for a given chunk on a single ISO. This data's footprint (equal
to the combined chunk sizes of the entire torrent) can be used for
nothing but this Spin. To be able to host 5 Spins composed from similar
trees via BitTorrent, we now have a footprint of 17GB, not to mention
"seeders" have to run BitTorrent software to be able to contribute to
the swarm. Alternatively, Jigdo can be used to reduce the footprint of
these 5 Spins to about 4GB. The amount of additional data needing to be
hosted for each Spin, in addition to what data is already pushed to the
mirrors, is about 150MB per ISO with anaconda and about 200KB for ISOs
without the installer bits. To help illustrate the efficiency of using
Jigdo vs BitTorrent, the footprint for 250 Spins is 850GB for
BitTorrent and about 40GB for Jigdo. Additionally, a reduction in
overhead can be achieved by removing the need for the BitTorrent
tracker and all related network traffic without requiring any
additional work on the part of mirror administrators.



My concern with jigdo is with how many people use it? It seems silly to
host both torrent and jigdo (as much of this letter points out the
benefits of switching to jigdo, those benefits disappear if we simply
add jigdo to the mix. Most people already have bittorrent. Lets say we
were going to give Jigdo a trial run for Fedora 9


FYI, we have done so, and we are doing so officially for Fedora 9.

and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?




Jigdo would in this case be particularly useful to those with a local
mirror as they have 99% of the content already (90% if you have F9T3?).
Because it is particularly useful to some, and completely weird and
strange for others, the number of users that will use it if BitTorrent
is an alternative wouldn't be a very good indicator to see if it is
actually a viable distribution method for the whole of Fedora, neither
is it the goal for these proposals.


However on the other hand we do have a couple of people with local
mirrors, and last time I checked, test releases are downloaded a couple
of times. We are hoping that these users in particular try out Jigdo and
become happy bandwidth and time savers.


For the Fedora Project, the greatest benefit of doing a trial Jigdo
release with Fedora 9 is to get to know the feeling, see the numbers,
get some feedback, and not having to host 68GB of Everything spins on
different media, instead of 450MB -giving the same results. The same
goes for any other additional installation media composed off the
release tree -even rebranded downstream distributions, although there
isn't much of those as long as updates keep expiring from the mirrors.
The original proposal was in fact that Fedora 9 CDs would be released
and hosted by Fedora Project but it seemed to be a better path to do so
via Release Engineering


While google trends is sort of crazy in that I don't know what it says,
it does say something:


http://google.com/trends?q=bittorrent%2C+jigdo



Right, I do hope our gut feeling rather then the number of hits on
Google comes up with a decision...


Another consideration in the entire footprint discussion may be to
expire FC{1,2,3,4,5,6} from the master mirror.


Kind regards,

Jeroen van Meeuwen
-kanarip

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:18 PM
Mike McGrath
 
Default Jigdo - A Professional Letter to Mike McGrath

Jeroen van Meeuwen wrote:

Mike McGrath wrote:

Jonathan Steffan wrote:

The amount of storage and bandwidth able to be saved can be illustrated
by a simple comparison between the efficiency of chopping up a 3.4GB
iso9660 file system arbitrarily (by a static chunk size) and the same
file system based on contents (file by file.) For a BitTorrent,
Fedora's current choice for sharing Spins, the hosted data is only
valid for a given chunk on a single ISO. This data's footprint (equal
to the combined chunk sizes of the entire torrent) can be used for
nothing but this Spin. To be able to host 5 Spins composed from similar
trees via BitTorrent, we now have a footprint of 17GB, not to mention
"seeders" have to run BitTorrent software to be able to contribute to
the swarm. Alternatively, Jigdo can be used to reduce the footprint of
these 5 Spins to about 4GB. The amount of additional data needing to be
hosted for each Spin, in addition to what data is already pushed to the
mirrors, is about 150MB per ISO with anaconda and about 200KB for ISOs
without the installer bits. To help illustrate the efficiency of using
Jigdo vs BitTorrent, the footprint for 250 Spins is 850GB for
BitTorrent and about 40GB for Jigdo. Additionally, a reduction in
overhead can be achieved by removing the need for the BitTorrent
tracker and all related network traffic without requiring any
additional work on the part of mirror administrators.



My concern with jigdo is with how many people use it? It seems silly
to host both torrent and jigdo (as much of this letter points out the
benefits of switching to jigdo, those benefits disappear if we simply
add jigdo to the mix. Most people already have bittorrent. Lets say
we were going to give Jigdo a trial run for Fedora 9


FYI, we have done so, and we are doing so officially for Fedora 9.

and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?




Jigdo would in this case be particularly useful to those with a local
mirror as they have 99% of the content already (90% if you have
F9T3?). Because it is particularly useful to some, and completely
weird and strange for others, the number of users that will use it if
BitTorrent is an alternative wouldn't be a very good indicator to see
if it is actually a viable distribution method for the whole of
Fedora, neither is it the goal for these proposals.


I'm talking specifically about people going to the get-fedora page and
clicking on the torrent link vs the jigdo link. Out of every 100
people, how many people will click on the jigdo link?


-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:25 PM
Mike McGrath
 
Default Jigdo - A Professional Letter to Mike McGrath

Jonathan Steffan wrote:

The amount of storage and bandwidth able to be saved can be illustrated
by a simple comparison between the efficiency of chopping up a 3.4GB
iso9660 file system arbitrarily (by a static chunk size) and the same
file system based on contents (file by file.) For a BitTorrent,
Fedora's current choice for sharing Spins, the hosted data is only
valid for a given chunk on a single ISO. This data's footprint (equal
to the combined chunk sizes of the entire torrent) can be used for
nothing but this Spin. To be able to host 5 Spins composed from similar
trees via BitTorrent, we now have a footprint of 17GB, not to mention
"seeders" have to run BitTorrent software to be able to contribute to
the swarm. Alternatively, Jigdo can be used to reduce the footprint of
these 5 Spins to about 4GB. The amount of additional data needing to be
hosted for each Spin, in addition to what data is already pushed to the
mirrors, is about 150MB per ISO with anaconda and about 200KB for ISOs
without the installer bits. To help illustrate the efficiency of using
Jigdo vs BitTorrent, the footprint for 250 Spins is 850GB for
BitTorrent and about 40GB for Jigdo. Additionally, a reduction in
overhead can be achieved by removing the need for the BitTorrent
tracker and all related network traffic without requiring any
additional work on the part of mirror administrators.



This paragraph shows the savings we would make on the jigdo server. How
much would our storage needs increase by needing to keep all RPM's
around on the mirrors?


-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:31 PM
Jeroen van Meeuwen
 
Default Jigdo - A Professional Letter to Mike McGrath

Mike McGrath wrote:

and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?




Jigdo would in this case be particularly useful to those with a local
mirror as they have 99% of the content already (90% if you have
F9T3?). Because it is particularly useful to some, and completely
weird and strange for others, the number of users that will use it if
BitTorrent is an alternative wouldn't be a very good indicator to see
if it is actually a viable distribution method for the whole of
Fedora, neither is it the goal for these proposals.


I'm talking specifically about people going to the get-fedora page and
clicking on the torrent link vs the jigdo link. Out of every 100
people, how many people will click on the jigdo link?




Given the choice to download, say, the Fedora 9 i386 vanilla DVD,
frankly, I expect only people that know Jigdo, or want to get to know
Jigdo as it may have some benefits for them, and want to use it, are
going to use it, so in all my optimism:


roughly 10 out of a 100.

For other spins without regular bittorrent seeds obviously the rate is
100%, and some of the people that get to know Jigdo that way will be
using it again for our respins, and if possible, again for other spins
(non-Everything?), and again, and again.


Kind regards,

Jeroen van Meeuwen
-kanarip

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:39 PM
Jeroen van Meeuwen
 
Default Jigdo - A Professional Letter to Mike McGrath

Mike McGrath wrote:
> This paragraph shows the savings we would make on the jigdo server.
How
much would our storage needs increase by needing to keep all RPM's
around on the mirrors?




For Fedora 8, per arch, at this moment that would be:

$ du -sch fedora-unity/releases/8/Everything/x86_64/
fedora-unity/updates/8/x86_64/

19G fedora-unity/releases/8/Everything/x86_64/
5.0G fedora-unity/updates/8/x86_64/
24G total

minus the updates in the normal tree already:

$ du -sch fedora/releases/8/Everything/x86_64/ fedora/updates/8/x86_64/
19G fedora/releases/8/Everything/x86_64/
3.9G fedora/updates/8/x86_64/
23G total

For Fedora 7, per arch, that'd be:

$ du -sch fedora-unity/releases/7/Everything/x86_64/
fedora-unity/updates/7/x86_64/

16G fedora-unity/releases/7/Everything/x86_64/
18G fedora-unity/updates/7/x86_64/
34G total

again minus the updates in the tree already:

$ du -sch fedora/releases/7/Everything/x86_64/ fedora/updates/7/x86_64/
16G fedora/releases/7/Everything/x86_64/
13G fedora/updates/7/x86_64/
28G total

So, roughly, 12GB per release per arch, not taking into account the ever
growing number of packages.


Kind regards,

Jeroen van Meeuwen
-kanarip

--
P.S. For everyone wondering, yes this host archives every single bit
that ever hits the mirrors except for development/ and releases/test/.


_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:52 PM
Mike McGrath
 
Default Jigdo - A Professional Letter to Mike McGrath

Chuck Anderson wrote:

On Fri, Dec 07, 2007 at 08:48:10AM -0600, Mike McGrath wrote:

My concern with jigdo is with how many people use it? It seems silly to
host both torrent and jigdo (as much of this letter points out the
benefits of switching to jigdo, those benefits disappear if we simply
add jigdo to the mix. Most people already have bittorrent. Lets say we
were going to give Jigdo a trial run for Fedora 9 and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?



Some people CAN'T use bittorrent because of firewalls. There should
be no reason at all why anyone couldn't use Jigdo, because it uses
standard FTP or HTTP to download the slices. There are clients
available for all the important OSes.



Those that can't use bittorrent can use the mirrors as well.

-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-07-2007, 02:53 PM
Chuck Anderson
 
Default Jigdo - A Professional Letter to Mike McGrath

On Fri, Dec 07, 2007 at 08:48:10AM -0600, Mike McGrath wrote:
> My concern with jigdo is with how many people use it? It seems silly to
> host both torrent and jigdo (as much of this letter points out the
> benefits of switching to jigdo, those benefits disappear if we simply
> add jigdo to the mix. Most people already have bittorrent. Lets say we
> were going to give Jigdo a trial run for Fedora 9 and we were going to
> judge jigdo a success if a certain % (compared to bittorrent) use
> jigdo. What % would that be?

Some people CAN'T use bittorrent because of firewalls. There should
be no reason at all why anyone couldn't use Jigdo, because it uses
standard FTP or HTTP to download the slices. There are clients
available for all the important OSes.

Jigdo and Bittorrent are really two different beasts that do different
things to benefit different use cases. Bittorrent is best for getting
all the bits for an ISO set when you have nothing currently. Jigdo is
good for getting bits that are packed differently but are otherwise
identical to the bits you have already, plus it can also get all the
bits via separate and possibly distributed downloads. So Jigdo in
this sense has a superset of the functionality of Bittorrent.

I think we need to have some overlap by providing both services,
certainly at least for a transition period, but perhaps even long
term. The real benefit to Jigdo is that you can distribute one set of
files that represent the Everthing universe of content via HTTP/FTP
(or ISOs via Bittorrent), and then a bunch of Jigdo templates for all
the various spins.

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-10-2007, 09:55 PM
Michael DeHaan
 
Default Jigdo - A Professional Letter to Mike McGrath

Jeroen van Meeuwen wrote:

Mike McGrath wrote:

and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?




Jigdo would in this case be particularly useful to those with a
local mirror as they have 99% of the content already (90% if you
have F9T3?). Because it is particularly useful to some, and
completely weird and strange for others, the number of users that
will use it if BitTorrent is an alternative wouldn't be a very good
indicator to see if it is actually a viable distribution method for
the whole of Fedora, neither is it the goal for these proposals.


I'm talking specifically about people going to the get-fedora page
and clicking on the torrent link vs the jigdo link. Out of every 100
people, how many people will click on the jigdo link?




Given the choice to download, say, the Fedora 9 i386 vanilla DVD,
frankly, I expect only people that know Jigdo, or want to get to know
Jigdo as it may have some benefits for them, and want to use it, are
going to use it, so in all my optimism:


roughly 10 out of a 100.


I would venture less. As a former Debian user (and knowing other
Debian users), our favorite install system was always the ~100 MB net
install image, which we can do for Fedora already (though it's not 100 MB)


Spins are perhaps interesting for users that don't do minimal
net-installs, and don't want to build their system with yum later, but
jigdo is something that advanced users would use. They seem to be two
different groups.


Jigdo did not seem to be very popular among anyone I talked to once they
figured out the minimal install images were available.


Fedora's MMV, of course.





_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-10-2007, 10:41 PM
Jeroen van Meeuwen
 
Default Jigdo - A Professional Letter to Mike McGrath

Michael DeHaan wrote:

Jeroen van Meeuwen wrote:

Mike McGrath wrote:

and we were going to
judge jigdo a success if a certain % (compared to bittorrent) use
jigdo. What % would that be?




Jigdo would in this case be particularly useful to those with a
local mirror as they have 99% of the content already (90% if you
have F9T3?). Because it is particularly useful to some, and
completely weird and strange for others, the number of users that
will use it if BitTorrent is an alternative wouldn't be a very good
indicator to see if it is actually a viable distribution method for
the whole of Fedora, neither is it the goal for these proposals.


I'm talking specifically about people going to the get-fedora page
and clicking on the torrent link vs the jigdo link. Out of every 100
people, how many people will click on the jigdo link?




Given the choice to download, say, the Fedora 9 i386 vanilla DVD,
frankly, I expect only people that know Jigdo, or want to get to know
Jigdo as it may have some benefits for them, and want to use it, are
going to use it, so in all my optimism:


roughly 10 out of a 100.


I would venture less. As a former Debian user (and knowing other
Debian users), our favorite install system was always the ~100 MB net
install image, which we can do for Fedora already (though it's not 100 MB)




Assuming you do have a network connection, like the example above; What
would you want to do if there's no Jigdo, but you do have the Fedora 9
DVD, and you want/need the CD version too? Download another ~4GB of ISO
images?


How about off-line?

It's fairly simple to create the Jigdo files and host them. Let's try it
and get the real numbers, then decide if it's valuable enough for all of
us to continue distributing installation media with.


Spins are perhaps interesting for users that don't do minimal
net-installs, and don't want to build their system with yum later, but
jigdo is something that advanced users would use. They seem to be two
different groups.


Jigdo did not seem to be very popular among anyone I talked to once they
figured out the minimal install images were available.




We on the other hand have hundreds -if not thousands- of users download
the CD version of Fedora 7 and Fedora 8 while supposedly they are in
possession of the DVD images already.


Kind regards,

Jeroen van Meeuwen
-kanarip

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 12-11-2007, 11:25 AM
Jesse Keating
 
Default Jigdo - A Professional Letter to Mike McGrath

On Tue, 11 Dec 2007 00:41:24 +0100
Jeroen van Meeuwen <kanarip@kanarip.com> wrote:

> We on the other hand have hundreds -if not thousands- of users
> download the CD version of Fedora 7 and Fedora 8 while supposedly
> they are in possession of the DVD images already.

What makes you say that? When I poked at fedoraunity.org the only
download option I saw was jigdo, so how are you determining that these
people are using jigdo by choice rather than by necessity?

--
Jesse Keating
Fedora -- All my bits are free, are yours?
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 

Thread Tools




All times are GMT. The time now is 05:57 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org