FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Development

 
 
LinkBack Thread Tools
 
Old 01-21-2009, 01:32 AM
Dan McGee
 
Default Arch mirror tracking woes

Whoopsies- wrong ML. Please reply here.


On Tue, Jan 20, 2009 at 3:36 PM, Dusty Phillips <buchuki@gmail.com> wrote:
> 2009/1/20 Aaron Griffin <aaronmgriffin@gmail.com>:
>> On Tue, Jan 20, 2009 at 12:04 AM, Geoffroy Carrier
>> <geoffroy.carrier@koon.fr> wrote:
>>> On Tue, Jan 20, 2009 at 05:24, Dan McGee <dpmcgee@gmail.com> wrote:
>>>> When you wake up, feel free to hack at this- it should do something
>>>> similar to what we want.
>>>
>>> http://github.com/gcarrier/automirror/
>>
>> Neat!
>>
>> The only thing I don't like... this is yet-another-mirror-list. We
>> currently have:
>> * pacman-mirrorlist
>> * The list of IPs for rsyncing mirrors
>> * A django DB of mirrors got the download page
>>
>> Any way you can reuse one of these?
>
> There's also a list of mirrors on the wiki. Its a pain to add a new
> mirror it has to be added in four places.

Let's get this stuff fixed. How does this look for a step in the right
direction?

class Mirror(models.Model):
id = models.AutoField(primary_key=True)
domain = models.CharField(max_length=255)
rsyncip = models.IPAddressField()
country = models.CharField(max_length=255)
admin_email = models.EmailField(max_length=255, null=True, blank=True)
def __unicode__(self):
return self.domain
class Meta:
db_table = 'mirrors'

class MirrorProtocol(models.Model):
id = models.AutoField(primary_key=True)
protocol = models.CharField(max_length=10, unique=True)

class MirrorUrl(models.Model):
id = models.AutoField(primary_key=True)
url = models.CharField(max_length=255)
protocol = models.ForeignKey(MirrorProtocol, related_name="urls")
mirror = models.ForeignKey(Mirror, related_name="urls")

1. rsync IP address will be stored right with the mirror. We can
easily run a DB query and script it to generate our list as needed for
our rsync configuration.
2. Mirror <-> URL is no longer a one-to-one mapping. A mirror could
have 1-4 URLs for any of the protocols it supports (FTP, HTTP, RSYNC,
and maybe one day HTTPS).
3. More detailed storage could turn into a Django view that says "give
me a HTTP and FTP mirrorlist for Canada" and boom! mirrorlist is made.
4. pacman-mirrorlist would be an automatically generated package if at
all possible. The caveat would be the current continent/country-based
sorting- any idea if we need to do something with the freeform Country
field to make this easier?
5. (CCed on this email) Gerhard, does this model look flexible enough
to perhaps incorporate mirror check status information in the future?
I would not mind seeing that feature on the main Arch site.
6. It might also be useful to add a "has_isos" boolean field to those
MirrorURLs (or Mirrors? not sure) that have ISOs mirrored. This would
ensure the download page makes sense.

Diff is below from the current model, although this may be quite a bit
less helpful than the above model.

diff --git a/main/models.py b/main/models.py
index b5a2a42..0fe17d5 100644
--- a/main/models.py
+++ b/main/models.py
@@ -52,15 +52,24 @@ class PackageManager(models.Manager):
class Mirror(models.Model):
id = models.AutoField(primary_key=True)
domain = models.CharField(max_length=255)
+ rsyncip = models.IPAddressField()
country = models.CharField(max_length=255)
- url = models.CharField(max_length=255)
- protocol_list = models.CharField(max_length=255, null=True, blank=True)
- admin_email = models.CharField(max_length=255, null=True, blank=True)
+ admin_email = models.EmailField(max_length=255, null=True, blank=True)
def __unicode__(self):
return self.domain
class Meta:
db_table = 'mirrors'

+class MirrorProtocol(models.Model):
+ id = models.AutoField(primary_key=True)
+ protocol = models.CharField(max_length=10, unique=True)
+
+class MirrorUrl(models.Model):
+ id = models.AutoField(primary_key=True)
+ url = models.CharField(max_length=255)
+ protocol = models.ForeignKey(MirrorProtocol, related_name="urls")
+ mirror = models.ForeignKey(Mirror, related_name="urls")
+
class Press(models.Model):
id = models.AutoField(primary_key=True)
name = models.CharField(max_length=255)
 
Old 01-21-2009, 05:57 AM
Pierre Schmitz
 
Default Arch mirror tracking woes

Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
> How does this look for a step in the right
> direction?

Looks good to me. I use a similar approach for the server stats page. Do you
think it would be possible to store those data in the mirrorlist package
somehow?

--

Pierre Schmitz


Clemens-August-Straße 76
53115 Bonn

Telefon 0228 9716608
Mobil 0160 95269831
Jabber pierre@jabber.archlinux.de
WWW http://www.archlinux.de
 
Old 01-22-2009, 12:37 AM
Dan McGee
 
Default Arch mirror tracking woes

On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
>> How does this look for a step in the right
>> direction?
>
> Looks good to me. I use a similar approach for the server stats page. Do you
> think it would be possible to store those data in the mirrorlist package
> somehow?

Can you point me to the page you are talking about?

And what do you mean by storing it in the mirrorlist package? The
intent here is to make the Django DB the canonical source of
information that will be disseminated out to each of the places that
need it- the downloads page, a mirrors page, and a way to generate the
pacman mirrorlist.

-Dan
 
Old 01-22-2009, 02:17 AM
Dan McGee
 
Default Arch mirror tracking woes

On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
> On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
>> Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
>>> How does this look for a step in the right
>>> direction?
>>
>> Looks good to me. I use a similar approach for the server stats page. Do you
>> think it would be possible to store those data in the mirrorlist package
>> somehow?
>
> Can you point me to the page you are talking about?
>
> And what do you mean by storing it in the mirrorlist package? The
> intent here is to make the Django DB the canonical source of
> information that will be disseminated out to each of the places that
> need it- the downloads page, a mirrors page, and a way to generate the
> pacman mirrorlist.

Some mostly-complete work on this stuff is on my branches:
http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/heads/working
http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/heads/working

-Dan
 
Old 01-22-2009, 10:29 AM
Pierre Schmitz
 
Default Arch mirror tracking woes

Am Donnerstag 22 Januar 2009 02:37:10 schrieb Dan McGee:
> Can you point me to the page you are talking about?

http://www.archlinux.de/?page=MirrorStatus

> And what do you mean by storing it in the mirrorlist package?

My intention was just something to make those information (support for http,
ftp, rsync; country etc.) available for everyone. So my first idea was to
store those additional information as comments in the mirrorlist.

--

Pierre Schmitz


Clemens-August-Straße 76
53115 Bonn

Telefon 0228 9716608
Mobil 0160 95269831
Jabber pierre@jabber.archlinux.de
WWW http://www.archlinux.de
 
Old 02-06-2009, 10:47 PM
Dusty Phillips
 
Default Arch mirror tracking woes

2009/1/21 Dan McGee <dpmcgee@gmail.com>:
> On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
>> On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
>>> Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
>>>> How does this look for a step in the right
>>>> direction?
>>>
>>> Looks good to me. I use a similar approach for the server stats page. Do you
>>> think it would be possible to store those data in the mirrorlist package
>>> somehow?
>>
>> Can you point me to the page you are talking about?
>>
>> And what do you mean by storing it in the mirrorlist package? The
>> intent here is to make the Django DB the canonical source of
>> information that will be disseminated out to each of the places that
>> need it- the downloads page, a mirrors page, and a way to generate the
>> pacman mirrorlist.
>
> Some mostly-complete work on this stuff is on my branches:
> http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/heads/working
> http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/heads/working

Hey all,

Sorry for dropping the ball on this one. I've merged Dan's changes and
am in the process of manually adding mirrors. I have entered all the
details into the admin except the rsync IPs. I would appreciate it if
someone would add those manually at some point. Then somebody can
write scripts to automatically generate the rsync whitelist and pacman
mirrorlist from this data. (Or did you do that already, Dan?)

In addition, I had to verify each of the mirrors to check if they have
ISOs. In doing that I discovered some of the mirrors in the current
database are broken. Specifically these ones:

http://mirror.vmmatrix.net/ArchLinux/ (China)
mirrors.jakimowicz.com (France)
ftp://ftp.parrswood.net/Mirrors/ftp.archlinux.org/ (UK)
ftp://saule.mintis.lt/pub/linux/ (Lithuania)

They should be removed from the mirrorlist and rsync whitelist if they
exist there, as I've already removed them from the DB.

I suspect there may be mirrors on the wiki page and/or in the pacman
mirror list that are not in the database. It would be nice to get them
all synced up.

Finally, there are a bunch of unofficial mirrors listed here:
http://wiki.archlinux.org/index.php/Mirrors#Unofficial_mirrors Is
there a reason we shouldn't make some/all of these official?? Some of
them may have been done already.
 
Old 02-06-2009, 11:17 PM
Dan McGee
 
Default Arch mirror tracking woes

On Fri, Feb 6, 2009 at 5:47 PM, Dusty Phillips <buchuki@gmail.com> wrote:
> 2009/1/21 Dan McGee <dpmcgee@gmail.com>:
>> On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
>>> On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
>>>> Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
>>>>> How does this look for a step in the right
>>>>> direction?
>>>>
>>>> Looks good to me. I use a similar approach for the server stats page. Do you
>>>> think it would be possible to store those data in the mirrorlist package
>>>> somehow?
>>>
>>> Can you point me to the page you are talking about?
>>>
>>> And what do you mean by storing it in the mirrorlist package? The
>>> intent here is to make the Django DB the canonical source of
>>> information that will be disseminated out to each of the places that
>>> need it- the downloads page, a mirrors page, and a way to generate the
>>> pacman mirrorlist.
>>
>> Some mostly-complete work on this stuff is on my branches:
>> http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/heads/working
>> http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/heads/working
>
> Hey all,
>
> Sorry for dropping the ball on this one. I've merged Dan's changes and
> am in the process of manually adding mirrors. I have entered all the
> details into the admin except the rsync IPs. I would appreciate it if
> someone would add those manually at some point. Then somebody can
> write scripts to automatically generate the rsync whitelist and pacman
> mirrorlist from this data. (Or did you do that already, Dan?)

I can try to do the rsync IPs this weekend, as well as making
scripts/etc. that spit out what we need. For the mirrorlist, we can
just make this a view on the public site- for the rsync IP list, we
might be able to make it a view on the dev site.

> In addition, I had to verify each of the mirrors to check if they have
> ISOs. In doing that I discovered some of the mirrors in the current
> database are broken. Specifically these ones:
>
> http://mirror.vmmatrix.net/ArchLinux/ (China)
> mirrors.jakimowicz.com (France)
> ftp://ftp.parrswood.net/Mirrors/ftp.archlinux.org/ (UK)
> ftp://saule.mintis.lt/pub/linux/ (Lithuania)
>
> They should be removed from the mirrorlist and rsync whitelist if they
> exist there, as I've already removed them from the DB.

Noted, and I'll try to remove them from the mirrorlist this weekend.

> I suspect there may be mirrors on the wiki page and/or in the pacman
> mirror list that are not in the database. It would be nice to get them
> all synced up.

Agreed. That was the point of this whole exercise.

> Finally, there are a bunch of unofficial mirrors listed here:
> http://wiki.archlinux.org/index.php/Mirrors#Unofficial_mirrors Is
> there a reason we shouldn't make some/all of these official?? Some of
> them may have been done already.

I think our only definition of "official" is "do they sync from
gerolde?". Not sure if that answers the question, but yeah.

-Dan
 

Thread Tools




All times are GMT. The time now is 05:44 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org