FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 01-27-2010, 08:50 AM
Roberto Ragusa
 
Default Suggestions (how to choose mirrors)

Matt Domsch wrote:
> Fedora has 244 public mirrors listed at the moment, and hundreds more
> private mirrors. We have no way, nor really any desire, to know in
> real time the load and network capacity of any connection between
> each mirror and each specific user. Perhaps Akamai does, but that's
> not a service we pay for.

Just an idea: instead of (or in addition to) "blind" planning,
based on net topology, geography, declared bandwith etc.,
yum could use an exploration approach:

1) choose a few good mirrors candidate
2) download one file from each of them (first file from
first mirror, second file from second mirror, ....)
3) gather speed statistics
4) reevaluate best mirrors according to statistics for the
remaining files

If the downloads are sorted by increasing size, you basically
use the small ones to sample the mirrors and make a good choice
for the big ones at the end of the list.

(doing many downloads in parallel would be the real plus,
so the slow and ugly mirror taking 1 minute for the 40kB file
will complete while the good mirrors are serving you the
kernel and openoffice.org)

This would also be automatically optimal for local mirrors.

Note that this is almost how P2P sharing works (just missing
the sub-file granularity).

--
Roberto Ragusa mail at robertoragusa.it
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 01-27-2010, 01:42 PM
James Antill
 
Default Suggestions (how to choose mirrors)

On Wed, 2010-01-27 at 10:50 +0100, Roberto Ragusa wrote:
> Matt Domsch wrote:
> > Fedora has 244 public mirrors listed at the moment, and hundreds more
> > private mirrors. We have no way, nor really any desire, to know in
> > real time the load and network capacity of any connection between
> > each mirror and each specific user. Perhaps Akamai does, but that's
> > not a service we pay for.
>
> Just an idea: instead of (or in addition to) "blind" planning,
> based on net topology, geography, declared bandwith etc.,
> yum could use an exploration approach:
>
> 1) choose a few good mirrors candidate
> 2) download one file from each of them (first file from
> first mirror, second file from second mirror, ....)
> 3) gather speed statistics
> 4) reevaluate best mirrors according to statistics for the
> remaining files

This is fine in theory, but there are a few problems. And more than a
few changes have to be made before we can do it.

> If the downloads are sorted by increasing size, you basically
> use the small ones to sample the mirrors and make a good choice
> for the big ones at the end of the list.

Except we got significant complaints when we did that sort, so I'm not
dying to do it again (even though I did it, and thought it was better).

> (doing many downloads in parallel would be the real plus,
> so the slow and ugly mirror taking 1 minute for the 40kB file
> will complete while the good mirrors are serving you the
> kernel and openoffice.org)

Pipelining is on the TODO, and will likely be done sometime this year.
The rest of it is blocked behind that.

> This would also be automatically optimal for local mirrors.

Unlikely, the problem with local mirrors is that it's often optimal to
download from them as we are atm. ... any attempt to use another mirror
is slower (and sometimes more expensive).
In fact my local mirror is often so fast that I seriously doubt whether
much improvement could be made due to pipelining etc. (which is not the
case with anything outside my network).

--
James Antill - james@fedoraproject.org
http://yum.baseurl.org/wiki/releases
http://yum.baseurl.org/wiki/whatsnew/3.2.26
http://yum.baseurl.org/wiki/YumMultipleMachineCaching
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 01-27-2010, 01:51 PM
Garrett Holmstrom
 
Default Suggestions (how to choose mirrors)

On Wed, Jan 27, 2010 at 3:50 AM, Roberto Ragusa <mail@robertoragusa.it> wrote:
> Just an idea: instead of (or in addition to) "blind" planning,
> based on net topology, geography, declared bandwith etc.,
> yum could use an exploration approach:
>
> 1) choose a few good mirrors candidate
> 2) download one file from each of them (first file from
> first mirror, second file from second mirror, ....)
> 3) gather speed statistics
> 4) reevaluate best mirrors according to statistics for the
> remaining files


You're basically describing yum-plugin-fastestmirror. Of course that
doesn't get one any sort of parallelism when downloading packages, but
it answers one of your complaints by rating actual data rates.

--
Garrett Holmstrom
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 01-27-2010, 02:35 PM
James Antill
 
Default Suggestions (how to choose mirrors)

On Wed, 2010-01-27 at 08:51 -0600, Garrett Holmstrom wrote:
> On Wed, Jan 27, 2010 at 3:50 AM, Roberto Ragusa <mail@robertoragusa.it> wrote:
> > Just an idea: instead of (or in addition to) "blind" planning,
> > based on net topology, geography, declared bandwith etc.,
> > yum could use an exploration approach:
> >
> > 1) choose a few good mirrors candidate
> > 2) download one file from each of them (first file from
> > first mirror, second file from second mirror, ....)
> > 3) gather speed statistics
> > 4) reevaluate best mirrors according to statistics for the
> > remaining files
>
>
> You're basically describing yum-plugin-fastestmirror. Of course that
> doesn't get one any sort of parallelism when downloading packages, but
> it answers one of your complaints by rating actual data rates.

fastestmirror does not do "data download measuring", which the above
implies. It measures latency (via. connect), which is often a good
substitute and has the advantage of being fast enough. But it can do
obviously wrong things, like ignore a local mirror that had a temporary
problem.

It also uses measured data for upto 10 days, by default.

So personally I'd prefer to just rely on MirrorManager. But saying all
that a lot of people swear by it, and CentOS (who don't use
MirrorManager) require it.

--
James Antill - james@fedoraproject.org
http://yum.baseurl.org/wiki/releases
http://yum.baseurl.org/wiki/whatsnew/3.2.26
http://yum.baseurl.org/wiki/YumMultipleMachineCaching
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-11-2010, 11:38 AM
Rahul Sundaram
 
Default Suggestions (how to choose mirrors)

On 01/27/2010 08:12 PM, James Antill wrote:
> On Wed, 2010-01-27 at 10:50 +0100, Roberto Ragusa wrote:
>
> If the downloads are sorted by increasing size, you basically
>> use the small ones to sample the mirrors and make a good choice
>> for the big ones at the end of the list.
>>
> Except we got significant complaints when we did that sort, so I'm not
> dying to do it again (even though I did it, and thought it was better).
>

yum install yum-plugin-download-order

Rahul
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 09:49 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org