FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 02-05-2008, 08:13 PM
"Leo "costela" Antunes"
 
Default Automatic mirror selection - feedback appreciated

Hi,

Thanks to the help of Adam Borowski we now got a working solution for an
automatic mirror selection scheme.

Please help us test this infrastructure so that I can convince the DSA
to add it to our servers or at least give us a delegation from a
debian.{net,org} subzone!

Again: I have no intention of supplanting any current structure, only
adding some more functionality to what we already have.


How to make it work:
--------------------

Simply change whatever mirror you're currently using on
/etc/apt/sources.list to:

<arch>-geomirror.debian.net

Where <arch> is the output of "dpkg --print-architecture".

Please make sure the DNS propagation has hit you, since the debian.net
aliases were all created earlier today.


How it works:
-------------

The <arch>-geomirror.d.n addresses are manually added CNAMEs to a
PowerDNS server that manages the zone geomirror.angband.pl (helpfully
loaned by Adam Borowski). This server is running pdns-backend-pipe,
which simply runs a script to determine the best mirror based on the
contents of Mirrors.masterlist[0], the IP that originated the request
and the requested architecture.

The current logic for selecting mirrors is very crude and only makes
sure the selected mirror is on the user's country and has the selected
arch, prioritizing mirrors that are called 'ftp.*.debian.org' and leafs
over push mirrors.
One possible improvement would be fetching information from DMC[1] to
help the prioritizing of hosts based on freshness (this could also help
avoid temporarily downed mirrors).

The source code for the backend script is available at[2].

The name 'geomirror' was picked for lack of a better suggestion. Please
feel free to give one!


Known problems:
--------------

Since it's based on CNAMEs, a mirror that works on a separate virtual
host than the server's default won't be usable by this scheme.

In a quick test scan (ignoring all non-related errors), about 84 of the
358 archive mirrors are affected by this issue, 9 of which are top-level
country mirrors (BR, CL, HK, IS, IT, PT, TR, UK and 2 of the 4
round-robins for CA).
Simply adding:
"ServerAlias *.geomirror.debian.org"
(or whatever the official address ends up being) to apache-based mirrors
should suffice, so I hope getting a delegation from DSA would be enough
to ask the mirror admins to add this alias to their mirrors. Please
correct me if I'm wrong and there's some other big problem I'm not seeing.

Another minor problem is that the use of some DNS resolver that's not on
the same country as the user (OpenDNS, for instance) will result in a
incorrect guess by the server.


Cheers

[0]
http://cvs.debian.org/*checkout*/webwml/english/mirror/Mirrors.masterlist
[1] http://www.de.debian.org/dmc/today/
[2] git://git.debian.org/~costela/mirror_picker.git

--
Leo "costela" Antunes
[insert a witty retort here]
 
Old 02-05-2008, 09:02 PM
Raphael Geissert
 
Default Automatic mirror selection - feedback appreciated

Hello,

Leo "costela" Antunes wrote:
>
> Another minor problem is that the use of some DNS resolver that's not on
> the same country as the user (OpenDNS, for instance) will result in a
> incorrect guess by the server.

Thanks for the clarification, I was about to say that it failed to detect my
country as USA instead of MX.

Besides that it seems to work great

PD. What do you think about adding a built-in list of addresses which 'work'
for OpenDNS so they are excluded and some kind of http redirection is used
instead?

>
>
> Cheers
>

Cheers,
Raphael


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-05-2008, 10:57 PM
Michal Čihař
 
Default Automatic mirror selection - feedback appreciated

Hi

Dne Tue, 05 Feb 2008 22:13:48 +0100
"Leo "costela" Antunes" <costela@debian.org> napsal(a):

> The <arch>-geomirror.d.n addresses are manually added CNAMEs to a
> PowerDNS server that manages the zone geomirror.angband.pl (helpfully
> loaned by Adam Borowski). This server is running pdns-backend-pipe,
> which simply runs a script to determine the best mirror based on the
> contents of Mirrors.masterlist[0], the IP that originated the request
> and the requested architecture.

Hmm, looks like mine server IP address is not detected correctly - first
resolution ended up with Japan mirror, second one with UK and finally
the third used correct Czech ;-). Not using any proxy DNS.

--
Michal Čihař | http://cihar.com | http://blog.cihar.com
 
Old 02-06-2008, 12:18 AM
Adam Borowski
 
Default Automatic mirror selection - feedback appreciated

On Wed, Feb 06, 2008 at 08:57:52AM +0900, Michal Čihař wrote:
^^^^^
> Dne Tue, 05 Feb 2008 22:13:48 +0100
> "Leo "costela" Antunes" <costela@debian.org> napsal(a):
>
> > The <arch>-geomirror.d.n addresses are manually added CNAMEs to a
> > PowerDNS server that manages the zone geomirror.angband.pl (helpfully
> > loaned by Adam Borowski). This server is running pdns-backend-pipe,
> > which simply runs a script to determine the best mirror based on the
> > contents of Mirrors.masterlist[0], the IP that originated the request
> > and the requested architecture.
>
> Hmm, looks like mine server IP address is not detected correctly - first
> resolution ended up with Japan mirror, second one with UK and finally
> the third used correct Czech ;-). Not using any proxy DNS.

geoip is not perfect, but a glance at your mail headers suggest that there
may be some confusion from another source.

Received: from mort.cihar.com (mort.cihar.com [82.208.50.189])
by liszt.debian.org
Tue, 5 Feb 2008 23:58:07 +0000 (UTC)
Received: from p1185-ipbfp704tokaisakaetozai.aichi.ocn.ne.jp ([125.207.87.185] helo=raptor.cic)
by mort.cihar.com
Wed, 06 Feb 2008 00:58:05 +0100
Received: from localhost ([127.0.0.1] helo=raptor.cic ident=nijel)
by raptor.cic
Wed, 06 Feb 2008 08:57:57 +0900

IPs, rev names and time zones seem to match. The machine you sent the mail
from seems to be definitely in Japan, and geoip says so as well. The mail
server, mort.cihar.com, looks Czechish (per its time zone and your name) --
again, geoip confirms that.

Assuming all machines use a nearby DNS, could you check again, using
"nslookup" or similar to avoid http proxies? Travelling/sshing to the other
side of the world means you may use a resolver all the way at home.
Accidentally mistaking a remote ssh session for a local one could be another
explanation.

--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-06-2008, 12:46 AM
Michal Čihař
 
Default Automatic mirror selection - feedback appreciated

Hi

On Wed, 6 Feb 2008 02:18:10 +0100
Adam Borowski <kilobyte@angband.pl> wrote:

> geoip is not perfect, but a glance at your mail headers suggest that there
> may be some confusion from another source.
>
> Received: from mort.cihar.com (mort.cihar.com [82.208.50.189])
> by liszt.debian.org
> Tue, 5 Feb 2008 23:58:07 +0000 (UTC)
> Received: from p1185-ipbfp704tokaisakaetozai.aichi.ocn.ne.jp ([125.207.87.185] helo=raptor.cic)
> by mort.cihar.com
> Wed, 06 Feb 2008 00:58:05 +0100
> Received: from localhost ([127.0.0.1] helo=raptor.cic ident=nijel)
> by raptor.cic
> Wed, 06 Feb 2008 08:57:57 +0900
>
> IPs, rev names and time zones seem to match. The machine you sent the mail
> from seems to be definitely in Japan, and geoip says so as well. The mail
> server, mort.cihar.com, looks Czechish (per its time zone and your name) --
> again, geoip confirms that.

This server was the place where I tried the lookup and it is really in
Czech. It has own DNS server which does not use any proxy and I flushed
it's cache before each attempt. Just tested it again to be sure:

# Flush of DNS server cache
nijel@mort$ host i386-geomirror.debian.net
i386-geomirror.debian.net is an alias for i386.geomirror.angband.pl.
i386.geomirror.angband.pl is an alias for ftp2.jp.debian.org.
ftp2.jp.debian.org has address 210.157.158.33
# Flush of DNS server cache
nijel@mort$ host i386-geomirror.debian.net
i386-geomirror.debian.net is an alias for i386.geomirror.angband.pl.
i386.geomirror.angband.pl is an alias for ftp.cz.debian.org.
ftp.cz.debian.org has address 195.113.161.73

Any further resolutions seems to go correctly to ftp.cz.d.o.

Okay let's try it also another way - ask directly your server without
anything on the way. It seems that last reply is somehow cached even
for different source addresses. If I do query first here in Japan and
then over there in Czech, I get few results pointing to Japan mirror
before it updates to Czech one. The same happens when I then start
resolving in Japan - I get again the Czech server on first attempts.
Log follows, the time difference 8 hours is time zone shift:

nijel@mort$ dig i386.geomirror.angband.pl @nurnen.angband.pl

; <<>> DiG 9.4.2 <<>> i386.geomirror.angband.pl @nurnen.angband.pl
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 24505
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;i386.geomirror.angband.pl. IN A

;; ANSWER SECTION:
i386.geomirror.angband.pl. 3600 IN CNAME ftp2.jp.debian.org.

;; Query time: 184 msec
;; SERVER: 212.160.145.27#53(212.160.145.27)
;; WHEN: Wed Feb 6 02:40:03 2008
;; MSG SIZE rcvd: 75

nijel@mort$ dig i386.geomirror.angband.pl @nurnen.angband.pl

; <<>> DiG 9.4.2 <<>> i386.geomirror.angband.pl @nurnen.angband.pl
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16974
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;i386.geomirror.angband.pl. IN A

;; ANSWER SECTION:
i386.geomirror.angband.pl. 3600 IN CNAME ftp.cz.debian.org.

;; Query time: 56 msec
;; SERVER: 212.160.145.27#53(212.160.145.27)
;; WHEN: Wed Feb 6 02:40:05 2008
;; MSG SIZE rcvd: 74

nijel@jp$ dig i386.geomirror.angband.pl @nurnen.angband.pl

; <<>> DiG 9.4.2 <<>> i386.geomirror.angband.pl @nurnen.angband.pl
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11125
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;i386.geomirror.angband.pl. IN A

;; ANSWER SECTION:
i386.geomirror.angband.pl. 3600 IN CNAME ftp.cz.debian.org.

;; Query time: 327 msec
;; SERVER: 212.160.145.27#53(212.160.145.27)
;; WHEN: Wed Feb 6 10:41:15 2008
;; MSG SIZE rcvd: 74

nijel@jp$ dig i386.geomirror.angband.pl @nurnen.angband.pl

; <<>> DiG 9.4.2 <<>> i386.geomirror.angband.pl @nurnen.angband.pl
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 18208
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;i386.geomirror.angband.pl. IN A

;; ANSWER SECTION:
i386.geomirror.angband.pl. 3600 IN CNAME ftp2.jp.debian.org.

;; Query time: 332 msec
;; SERVER: 212.160.145.27#53(212.160.145.27)
;; WHEN: Wed Feb 6 10:42:31 2008
;; MSG SIZE rcvd: 75


--
Michal Čihař | http://cihar.com | http://blog.cihar.com
 
Old 02-06-2008, 01:13 PM
"Leo "costela" Antunes"
 
Default Automatic mirror selection - feedback appreciated

Michal Čihař wrote:
> This server was the place where I tried the lookup and it is really in
> Czech. It has own DNS server which does not use any proxy and I flushed
> it's cache before each attempt. Just tested it again to be sure:

[snip]

> Okay let's try it also another way - ask directly your server without
> anything on the way. It seems that last reply is somehow cached even
> for different source addresses. If I do query first here in Japan and
> then over there in Czech, I get few results pointing to Japan mirror
> before it updates to Czech one. The same happens when I then start
> resolving in Japan - I get again the Czech server on first attempts.
> Log follows, the time difference 8 hours is time zone shift:

Are you sure there's no redirection of port 53 going on there? Or
perhaps a load-balancing between two separate outbound IPs (one of which
could be wrong in the GeoIP database)? I don't mean to say you don't
know how your servers work, it's just that this seems a bit odd, given
that nobody else reported similar problems and how simple the setup is.

I don't think it's a cache problem on PDNS, firstly because it doesn't
seem to be caching anything (based on log output) and secondly because
the answer is changing between requests inside a small time-frame. But I
could be wrong.

Can you hop on to IRC to coordinate some queries while I watch the log?


Cheers

--
Leo "costela" Antunes
[insert a witty retort here]
 
Old 02-06-2008, 03:56 PM
"Leo "costela" Antunes"
 
Default Automatic mirror selection - feedback appreciated

[setting MFT to avoid OT discussion on -mirror]

Raphael Geissert wrote:
> PD. What do you think about adding a built-in list of addresses which 'work'
> for OpenDNS so they are excluded and some kind of http redirection is used
> instead?

It's a possibility, but I had envisioned something simpler. IMHO this
falls under the "user knows better than to use an automated system"
scenario, for people that would probably be picking their mirrors
manually anyway. I don't know of any ISP that uses off-shore DNS
resolvers, so people that do this are probably already changing it manually.
But of course, I'm open for discussion if people think this is a worthy
workaround (and you're also free get the script yourself and provide
some patches ).

Cheers and thanks for the feedback!

--
Leo "costela" Antunes
[insert a witty retort here]
 
Old 02-13-2008, 03:43 PM
"Leo "costela" Antunes"
 
Default Automatic mirror selection - feedback appreciated

Leo "costela" Antunes wrote:
> I don't think it's a cache problem on PDNS, firstly because it doesn't
> seem to be caching anything (based on log output) and secondly because
> the answer is changing between requests inside a small time-frame. But I
> could be wrong.

Thanks to the help of Michal Cihar this problem has been fixed.

If anyone is still interested, please help by running a few more test
'apt-get update && apt-get upgrade', for instance.

Cheers

--
Leo "costela" Antunes
[insert a witty retort here]
 

Thread Tools




All times are GMT. The time now is 07:38 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org