Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian Development (http://www.linux-archive.org/debian-development/)
-   -   Automatic mirror selection (http://www.linux-archive.org/debian-development/44284-automatic-mirror-selection.html)

"Leo "costela" Antunes" 02-01-2008 12:38 AM

Automatic mirror selection
 
Hey there,

As an incredibly late follow-up to this [0] small thread, I created a
small script to act as backend for pdns and return a mirror for the
user's country.
It's a simple DNS based geographic mirror selection idea.

It works by:
- using logic based on D-I to select a mirror from a copy of
Mirrors.masterlist[1], making it behave like a user that selects his own
country during installation.
- filtering by country and arch, with a fallback host if the country
isn't found or no mirrors provide the needed arch.
- applying a _very_ simple priority scheme to the mirrors that match,
giving top points to hosts that match "ftp{1,2}.{2}.debian.org" and also
preferring "leaf" over "push" mirrors.

This last point is something I am still reluctant about: the logic was
"leafs will tend to be less loaded", but this is really not true;
perhaps some priority like "secondary > primary > leaf", to offload
primaries, but keep leafs as a last resort would be better.

I'd like to put this to use for Debian, but I face two small problems:
- I currently don't have access to any server in which I could host this.
- our d.net domains apparently[2] can't contain NS records, which means
I couldn't have anything more "integrated" without DSA's approval, even
if I had a machine.

So, which friendly soul could I ask about getting this running on a
Debian server with a d.net domain, assuming there's some interest in it
aside from my own? The DSA through a ticket?


Please fire away if you have any comments on the idea, but keep in mind
it's not intended to supplant anything we currently have and it's
obviously not intended for every scenario.


Cheers

[0] http://thread.gmane.org/gmane.linux.debian.user.mirrors/311
[1]
http://cvs.debian.org/*checkout*/webwml/english/mirror/Mirrors.masterlist
[2] http://db.debian.org/doc-mail.html

--
Leo "costela" Antunes
[insert a witty retort here]

Adam Borowski 02-01-2008 08:40 AM

Automatic mirror selection
 
On Fri, Feb 01, 2008 at 02:38:04AM +0100, Leo costela Antunes wrote:
> Hey there,
>
> As an incredibly late follow-up to this [0] small thread, I created a
> small script to act as backend for pdns and return a mirror for the
> user's country.
> It's a simple DNS based geographic mirror selection idea.

Hell yeah. Finally.

We got plenty of packages which pull debs from their author's country:
ftp.pl.debian.org, ftp.uk.debian.org. It would be helluva better to have
them use geoip by default.

I once gave this a though, but never went any farther than checking which
mirrors work. The status page is still online:
http://angband.pl/deb/mirrors
although it doesn't even update the mirror list. There are better such
status pages elsewhere, too, except none checks IPv4 and IPv6 separately.

> It works by:
> - using logic based on D-I to select a mirror from a copy of
> Mirrors.masterlist[1], making it behave like a user that selects his own
> country during installation.

Since the packaged version of geoip gives you just the country, my idea
would be to simply give a cname for ftp.XX.debian.org. On MaxMind's page,
you can also get a "geoip-city-lite" database which appears to be dfsg free
once you purge nondistributable parts about ISPs (city and ISP data is
intermingled in a single file). That could perhaps provide some better
resolution than just country.

> - filtering by country and arch, with a fallback host if the country
> isn't found or no mirrors provide the needed arch.

Uhm, and how exactly do you get the arch? At DNS time you don't have
anything but the requester's or his ISP's IP.

> I'd like to put this to use for Debian, but I face two small problems:
> - I currently don't have access to any server in which I could host this.

Would a vserver, one IP and a NS delegation for a test subdomain be enough
for your tests? No fallover or anything, though.

> - our d.net domains apparently[2] can't contain NS records, which means
> I couldn't have anything more "integrated" without DSA's approval, even
> if I had a machine.

What about this:
ftptest.debian.net CNAME debian.XXX
debian.XXX NS your_vserver.XXX
(where XXX is any domain you can control without bothering the DSA for every
change)

> So, which friendly soul could I ask about getting this running on a
> Debian server with a d.net domain, assuming there's some interest in it
> aside from my own? The DSA through a ticket?

This would be a great addition to the official infrastructure. If this goes
official, it would need some reliable hosting, but for tests, you can use
any machine, even one of mine.

> Please fire away if you have any comments on the idea, but keep in mind
> it's not intended to supplant anything we currently have and it's
> obviously not intended for every scenario.

To the contrary, with the name "ftp.debian.org" being deprecated, I can
think of a perfect case you can supplant once you go official :p


Great work, carry on!

--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

"Leo "costela" Antunes" 02-01-2008 07:09 PM

Automatic mirror selection
 
Adam Borowski wrote:
> you can also get a "geoip-city-lite" database which appears to be dfsg free
> once you purge nondistributable parts about ISPs (city and ISP data is
> intermingled in a single file). That could perhaps provide some better
> resolution than just country.

I thougth about this too, but we don't currently have reliable city
information for the mirrors, so it would demand more work and wouldn't
bring big benefits, since it's already not certain that the user's
country's mirror will actually be any faster than an inter-country link,
I decided that going for a country based approach was the best
middle-ground bet. We'd have even less certainty about the user's city's
mirror performance.

> Uhm, and how exactly do you get the arch? At DNS time you don't have
> anything but the requester's or his ISP's IP.

This would have to be placed inside sources.list at some point, then I
figure the automatic or manual process responsible should stick
"dpkg-architecture -qDEB_BUILD_ARCH" somewhere in there, so you just
have to query "<arch>.geomirror.d.net", for instance, to get the best
mirror for your country which contains your arch.
It could be during D-I, if the D-I people like the idea and no big
problems arise from a testing period.

> Would a vserver, one IP and a NS delegation for a test subdomain be enough
> for your tests? No fallover or anything, though.

Yes, certainly. Do you have one available?
The only needed things are:
- pdns-backend-pipe
- libgeo-ip-perl


> What about this:
> ftptest.debian.net CNAME debian.XXX
> debian.XXX NS your_vserver.XXX
> (where XXX is any domain you can control without bothering the DSA for every
> change)

I don't think that'll work. Since the Debian nameserver won't find a
break in the SOA line (introduced with a NS entry), it'll try to find
YYY.ftptest.d.net directly. A CNAME entry on an upper-level domain won't
be enough to "redirect" the lower-level domains.
But I'm no DNS specialist, so if you (or anyone) have a more specific
way of making this work, I'm all ears!



Thanks for the feedback!

Cheers

--
Leo "costela" Antunes
[insert a witty retort here]

Adam Borowski 02-01-2008 08:32 PM

Automatic mirror selection
 
On Fri, Feb 01, 2008 at 09:09:24PM +0100, Leo costela Antunes wrote:
> Adam Borowski wrote:
> > Uhm, and how exactly do you get the arch? At DNS time you don't have
> > anything but the requester's or his ISP's IP.
>
> This would have to be placed inside sources.list at some point, then I
> figure the automatic or manual process responsible should stick
> "dpkg-architecture -qDEB_BUILD_ARCH" somewhere in there, so you just
> have to query "<arch>.geomirror.d.net", for instance, to get the best
> mirror for your country which contains your arch.
> It could be during D-I, if the D-I people like the idea and no big
> problems arise from a testing period.

Ah, right. Yet at least in some cases, it could be tricky to have the arch,
so I think you would need a generic mirror anyway.

> > Would a vserver, one IP and a NS delegation for a test subdomain be enough
> > for your tests? No fallover or anything, though.
>
> Yes, certainly. Do you have one available?

(Details sent privately). Too bad, I just learned it can possibly go down
somewhere in the near future...

> > What about this:
> > ftptest.debian.net CNAME debian.XXX
> > debian.XXX NS your_vserver.XXX
> > (where XXX is any domain you can control without bothering the DSA for every
> > change)
>
> I don't think that'll work. Since the Debian nameserver won't find a
> break in the SOA line (introduced with a NS entry), it'll try to find
> YYY.ftptest.d.net directly. A CNAME entry on an upper-level domain won't
> be enough to "redirect" the lower-level domains.

Aargh, it appears you're right. Does someone have another idea?

If not, we can always do a brief test on a random subdomain, and then toss a
working config to the DSA guys.


Cheers, and so on...
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

"Leo "costela" Antunes" 02-02-2008 12:10 AM

Automatic mirror selection
 
Adam Borowski wrote:
> Ah, right. Yet at least in some cases, it could be tricky to have the arch,
> so I think you would need a generic mirror anyway.

I don't see any scenario where getting the arch would be tricky, do you
have any in mind?
Even if there is such a case, we have many mirrors that don't support
all arches, so - to avoid problems - we'd have to filter the list to
have only complete mirrors, drastically diminishing its utility,
therefore I think it would be best to ignore the very few - if any -
corner cases in which it's a little harder to get the arch, in the name
of maintained utility.

> (Details sent privately). Too bad, I just learned it can possibly go down
> somewhere in the near future...

Thanks a lot anyway!

> If not, we can always do a brief test on a random subdomain, and then toss a
> working config to the DSA guys.

Cheers

--
Leo "costela" Antunes
[insert a witty retort here]

Raphael Geissert 02-03-2008 09:09 PM

Automatic mirror selection
 
Hello,

Leo "costela" Antunes wrote:

> Hey there,
>
> As an incredibly late follow-up to this [0] small thread, I created a
> small script to act as backend for pdns and return a mirror for the
> user's country.
> It's a simple DNS based geographic mirror selection idea.
>
> It works by:
> - using logic based on D-I to select a mirror from a copy of
> Mirrors.masterlist[1], making it behave like a user that selects his own
> country during installation.
> - filtering by country and arch, with a fallback host if the country
> isn't found or no mirrors provide the needed arch.
> - applying a _very_ simple priority scheme to the mirrors that match,
> giving top points to hosts that match "ftp{1,2}.{2}.debian.org" and also
> preferring "leaf" over "push" mirrors.

I'm not an expert on the subject, but wouldn't anycast be more suitable?

>
>
>
> Cheers
>

Cheers,
Raphael Geissert



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

"Leo "costela" Antunes" 02-03-2008 10:01 PM

Automatic mirror selection
 
Hi,

Raphael Geissert wrote:
>
> I'm not an expert on the subject, but wouldn't anycast be more suitable?
>

It's been suggested on the referred thread[0].
I've never worked with it myself, but my limited understanding is that
it would need work from all mirror admins and even if that was feasible,
there are bound to be mirrors that are (technically, bureaucratically,
etc) unable to perform the needed changes to integrate their
infrastructure with anycast.


Cheers

[0] http://thread.gmane.org/gmane.linux.debian.user.mirrors/311/focus=312

--
Leo "costela" Antunes
[insert a witty retort here]

Raphael Geissert 02-04-2008 12:29 AM

Automatic mirror selection
 
Hi,

Leo "costela" Antunes wrote:
>
> It's been suggested on the referred thread[0].
> I've never worked with it myself, but my limited understanding is that
> it would need work from all mirror admins and even if that was feasible,
> there are bound to be mirrors that are (technically, bureaucratically,
> etc) unable to perform the needed changes to integrate their
> infrastructure with anycast.

Seems like Simon Blake covered the main points very well so there isn't very
much left I can say.

Besides that I'm interested in how your script works at DNS level and if it
wouldn't be more suitable to just setup a server with BIND + GeoDNS[1].


>
>
> Cheers
>
> [0] http://thread.gmane.org/gmane.linux.debian.user.mirrors/311/focus=312
>

Cheers,
Raphael

[1]http://www.caraytech.com/geodns/




--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Guillem Jover 02-04-2008 06:27 PM

Automatic mirror selection
 
On Fri, 2008-02-01 at 21:09:24 +0100, Leo costela Antunes wrote:
> Adam Borowski wrote:
> > Uhm, and how exactly do you get the arch? At DNS time you don't have
> > anything but the requester's or his ISP's IP.
>
> This would have to be placed inside sources.list at some point, then I
> figure the automatic or manual process responsible should stick
> "dpkg-architecture -qDEB_BUILD_ARCH" somewhere in there, so you just
> have to query "<arch>.geomirror.d.net", for instance, to get the best
> mirror for your country which contains your arch.

You might want you use «dpkg --print-architecture» instead, so you
avoid a dependency on dpkg-dev.

regards,
guillem


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

"Leo "costela" Antunes" 02-04-2008 07:17 PM

Automatic mirror selection
 
Guillem Jover wrote:
> You might want you use «dpkg --print-architecture» instead, so you
> avoid a dependency on dpkg-dev.

Good point.

Cheers

--
Leo "costela" Antunes
[insert a witty retort here]


All times are GMT. The time now is 07:23 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.