FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 04-24-2012, 01:10 PM
Corentin Chary
 
Default New third party mirrors

Hello,

As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
I created a tool that counts URIs prefixes and sort them.
Here is the result of this script on the gentoo-x86 tree:

$ ./mirrors.py --all --count
1262 http://dev.gentoo.org
483 http://oss.tresys.com
482 http://oss.tresys.com/files
482 http://oss.tresys.com/files/refpolicy
473 http://xorg.freedesktop.org
473 http://xorg.freedesktop.org/releases
473 http://xorg.freedesktop.org/releases/individual
313 http://dev.gentoo.org/~swift
313 http://dev.gentoo.org/~swift/patches
309 http://dev.gentoo.org/~swift/patches/selinux-base-policy
297 http://pear.php.net
297 http://pear.php.net/get
296 http://hackage.haskell.org/packages
296 http://hackage.haskell.org/packages/archive
296 http://hackage.haskell.org
209 http://launchpad.net
206 http://ftp.xemacs.org
197 https://github.com
196 http://ftp.xemacs.org/pub
196 http://ftp.xemacs.org/pub/xemacs
193 http://ftp.xemacs.org/pub/xemacs/packages
179 http://gstreamer.freedesktop.org
179 http://gstreamer.freedesktop.org/src
176 http://github.com
175 http://linuxgazette.net/ftpfiles
175 http://linuxgazette.net
139 http://xorg.freedesktop.org/releases/individual/app
130 http://pear.horde.org/get
130 http://pear.horde.org
126 http://xorg.freedesktop.org/releases/individual/driver
115 http://dev.gentoo.org/~vapier
112 http://dev.gentoo.org/~vapier/dist
96 ftp://sources.redhat.com/pub
96 ftp://sources.redhat.com
94 http://savannah.nongnu.org/download
94 http://savannah.nongnu.org
92 http://xorg.freedesktop.org/releases/individual/lib
88 http://pecl.php.net
88 http://download.gna.org
88 http://pecl.php.net/get
84 http://dev.gentoo.org/~caster/distfiles
84 http://dev.gentoo.org/~caster
80 https://fedorahosted.org
74 http://get.qt.nokia.com
72 http://components.ez.no/get
72 http://components.ez.no
71 http://get.qt.nokia.com/qt
70 http://get.qt.nokia.com/qt/source
67 http://www.phrack.org/archives
67 http://www.phrack.org/archives/tgz

And here are some mirrors that can be deduced from that (except freebsd-jp
which I found reading the linphone ebuild):

freebsd-jp ftp://ports.jp.FreeBSD.org/pub/FreeBSD-jp/ ftp://ftp.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp1.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp2.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp3.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp4.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp5.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp6.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp7.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp8.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp9.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/mirrors/FreeBSD-jp/ ftp://ftp.allbsd.org/pub/FreeBSD-jp/ ftp://ftp.nagaokaut.ac.jp/pub/FreeBSD-jp/ ftp://ftp.nara.wide.ad.jp/pub/FreeBSD-jp/ ftp://ftp.jaist.ac.jp/pub/FreeBSD-jp/
pear http://pear.php.net/ http://de.pear.php.net/ http://us.pear.php.net/
redhat ftp://sources.redhat.com/pub/ ftp://mirrors.kernel.org/sources.redhat.com/ ftp://mirrors.kernel.org/sources.redhat.com/ http://mirrors.kernel.org/sources.redhat.com/ http://sources-redhat.mirrors.airband.net/ ftp://gd.tuwien.ac.at/gnu/sourceware/ http://gd.tuwien.ac.at/gnu/sourceware/ ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/ http://ftp.gwdg.de/pub/linux/sources.redhat.com/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://ftp.mirrorservice.org/sites/sources.redhat.com/pub/ http://www.mirrorservice.org/sites/sources.redhat.com/pub/
xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
xorg http://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/

That would affect ~1000 URIs.
Unfortunatly I didn't found any mirror for:
- pecl.php.net
- hackage.haskell.org
- get.qt.nokia.com
- launchpad

But maybe we should still add them to thirdpartymirrors ? Look at 3dgamers, beyondunreal, bitbucket, github, gnome, kernel, xfce, vdrfiles and vdr-developerorg: they all only have one mirror !

For those afraid to add these mirrors and patch all affected ebuilds: no worry, mirrors.py can do it for you, and I already generated splitted patches generated from various thirdpartymirrors here: http://xf.iksaif.net/dev/gentoo/

Note that as part of https://bugs.gentoo.org/show_bug.cgi?id=405533 mgorny already commited a massive amount of patches adding mirror:// when the URI prefix was already present in thirdpartymirrors.

Thanks,

--
Corentin Chary
http://xf.iksaif.net/
 
Old 04-24-2012, 02:35 PM
"Robin H. Johnson"
 
Default New third party mirrors

On Tue, Apr 24, 2012 at 03:10:00PM +0200, Corentin Chary wrote:
> Hello,
>
> As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
> I created a tool that counts URIs prefixes and sort them.
> Here is the result of this script on the gentoo-x86 tree:
>
> $ ./mirrors.py --all --count
> 297 http://pear.php.net
> 297 http://pear.php.net/get
> 88 http://pecl.php.net
> 88 http://pecl.php.net/get
These are already mirror bouncers. If you visit the above, you'll get
the closest mirror for downloading.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
 
Old 04-24-2012, 02:50 PM
Corentin Chary
 
Default New third party mirrors

On Tue, Apr 24, 2012 at 4:35 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Tue, Apr 24, 2012 at 03:10:00PM +0200, Corentin Chary wrote:
>> Hello,
>>
>> As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
>> I created a tool that counts URIs prefixes and sort them.
>> Here is the result of this script on the gentoo-x86 tree:
>>
>> $ ./mirrors.py --all --count
>> 297 * * http://pear.php.net
>> 297 * * http://pear.php.net/get
>> 88 * * *http://pecl.php.net
>> 88 * * *http://pecl.php.net/get
> These are already mirror bouncers. If you visit the above, you'll get
> the closest mirror for downloading.

And since there is already ~10 "mirrors" with only one actual backend,
should they go to thirdpartymirrors or not ? If not, what about this
pseudo-mirrors already present in thirdpartymirrors ?

--
Corentin Chary
http://xf.iksaif.net
 
Old 04-24-2012, 04:19 PM
"Robin H. Johnson"
 
Default New third party mirrors

On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> >> $ ./mirrors.py --all --count
> >> 297 ?? ?? http://pear.php.net
> >> 297 ?? ?? http://pear.php.net/get
> >> 88 ?? ?? ??http://pecl.php.net
> >> 88 ?? ?? ??http://pecl.php.net/get
> > These are already mirror bouncers. If you visit the above, you'll get
> > the closest mirror for downloading.
> And since there is already ~10 "mirrors" with only one actual backend,
> should they go to thirdpartymirrors or not ? If not, what about this
> pseudo-mirrors already present in thirdpartymirrors ?
I think we should add the pseudo-mirrors, but explicitly mark them as
such in the file, so that they don't get duplicate entries added (eg
adding us.pear, de.pear and the pear bouncer is bad. Should have just
the bouncer).

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
 
Old 04-24-2012, 04:38 PM
Michał Górny
 
Default New third party mirrors

On Tue, 24 Apr 2012 16:19:11 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> > >> $ ./mirrors.py --all --count
> > >> 297 ?? ?? http://pear.php.net
> > >> 297 ?? ?? http://pear.php.net/get
> > >> 88 ?? ?? ??http://pecl.php.net
> > >> 88 ?? ?? ??http://pecl.php.net/get
> > > These are already mirror bouncers. If you visit the above, you'll
> > > get the closest mirror for downloading.
> > And since there is already ~10 "mirrors" with only one actual
> > backend, should they go to thirdpartymirrors or not ? If not, what
> > about this pseudo-mirrors already present in thirdpartymirrors ?
> I think we should add the pseudo-mirrors, but explicitly mark them as
> such in the file, so that they don't get duplicate entries added (eg
> adding us.pear, de.pear and the pear bouncer is bad. Should have just
> the bouncer).

It'd be great if we could add some kind of additional mirror entries,
which would be used by repoman to signal missing mirror:// entries but
won't be used for downloads.

--
Best regards,
Michał Górny
 
Old 04-24-2012, 04:49 PM
Zac Medico
 
Default New third party mirrors

On 04/24/2012 09:38 AM, Michał Górny wrote:

On Tue, 24 Apr 2012 16:19:11 +0000
"Robin H. Johnson"<robbat2@gentoo.org> wrote:


On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:

$ ./mirrors.py --all --count
297 ?? ?? http://pear.php.net
297 ?? ?? http://pear.php.net/get
88 ?? ?? ??http://pecl.php.net
88 ?? ?? ??http://pecl.php.net/get

These are already mirror bouncers. If you visit the above, you'll
get the closest mirror for downloading.

And since there is already ~10 "mirrors" with only one actual
backend, should they go to thirdpartymirrors or not ? If not, what
about this pseudo-mirrors already present in thirdpartymirrors ?

I think we should add the pseudo-mirrors, but explicitly mark them as
such in the file, so that they don't get duplicate entries added (eg
adding us.pear, de.pear and the pear bouncer is bad. Should have just
the bouncer).


It'd be great if we could add some kind of additional mirror entries,
which would be used by repoman to signal missing mirror:// entries but
won't be used for downloads.


We could just add them in thirpartymirrors under fake names that nobody
will use.

--
Thanks,
Zac
 
Old 04-24-2012, 05:54 PM
Hans de Graaff
 
Default New third party mirrors

On Tue, 2012-04-24 at 15:10 +0200, Corentin Chary wrote:

> And here are some mirrors that can be deduced from that (except freebsd-jp
> which I found reading the linphone ebuild):
>
> xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/

I only consider the first site to be a reliable source for the packages,
and this is what the eclass provides (only 4 packages actually provide a
SRC_URI, all others are handled through the eclass).

Kind regards,

Hans
 
Old 04-25-2012, 07:16 AM
Corentin Chary
 
Default New third party mirrors

On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Tue, 24 Apr 2012 16:19:11 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>
>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>> > >> $ ./mirrors.py --all --count
>> > >> 297 ?? ?? http://pear.php.net
>> > >> 297 ?? ?? http://pear.php.net/get
>> > >> 88 ?? ?? ??http://pecl.php.net
>> > >> 88 ?? ?? ??http://pecl.php.net/get
>> > > These are already mirror bouncers. If you visit the above, you'll
>> > > get the closest mirror for downloading.
>> > And since there is already ~10 "mirrors" with only one actual
>> > backend, should they go to thirdpartymirrors or not ? If not, what
>> > about this pseudo-mirrors already present in thirdpartymirrors ?
>> I think we should add the pseudo-mirrors, but explicitly mark them as
>> such in the file, so that they don't get duplicate entries added (eg
>> adding us.pear, de.pear and the pear bouncer is bad. Should have just
>> the bouncer).
>
> It'd be great if we could add some kind of additional mirror entries,
> which would be used by repoman to signal missing mirror:// entries but
> won't be used for downloads.

Yep, we could put that in it too:
github http://github.com/downloads/ https://github.com/downloads/
nongnu http://download.savannah.nongnu.org/releases/

(I'm not really sure for nongnu).


--
Corentin Chary
http://xf.iksaif.net
 
Old 04-25-2012, 04:41 PM
Michał Górny
 
Default New third party mirrors

On Wed, 25 Apr 2012 09:16:05 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > On Tue, 24 Apr 2012 16:19:11 +0000
> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> >
> >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> >> > >> $ ./mirrors.py --all --count
> >> > >> 297 ?? ?? http://pear.php.net
> >> > >> 297 ?? ?? http://pear.php.net/get
> >> > >> 88 ?? ?? ??http://pecl.php.net
> >> > >> 88 ?? ?? ??http://pecl.php.net/get
> >> > > These are already mirror bouncers. If you visit the above,
> >> > > you'll get the closest mirror for downloading.
> >> > And since there is already ~10 "mirrors" with only one actual
> >> > backend, should they go to thirdpartymirrors or not ? If not,
> >> > what about this pseudo-mirrors already present in
> >> > thirdpartymirrors ?
> >> I think we should add the pseudo-mirrors, but explicitly mark them
> >> as such in the file, so that they don't get duplicate entries
> >> added (eg adding us.pear, de.pear and the pear bouncer is bad.
> >> Should have just the bouncer).
> >
> > It'd be great if we could add some kind of additional mirror
> > entries, which would be used by repoman to signal missing mirror://
> > entries but won't be used for downloads.
>
> Yep, we could put that in it too:
> github http://github.com/downloads/
> https://github.com/downloads/

Per spec, portage can choose a random mirror of the list. If we put
entries like that, these two will be equally possible as the preferred
cloud. URL -- while they redirect one to another.

We might decide on some common syntax like preceding all extra entries
with '-' but I don't want to be the one deciding here.

--
Best regards,
Michał Górny
 
Old 04-26-2012, 07:30 AM
Corentin Chary
 
Default New third party mirrors

On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Wed, 25 Apr 2012 09:16:05 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>> wrote:
>> > On Tue, 24 Apr 2012 16:19:11 +0000
>> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>> >
>> >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>> >> > >> $ ./mirrors.py --all --count
>> >> > >> 297 ?? ?? http://pear.php.net
>> >> > >> 297 ?? ?? http://pear.php.net/get
>> >> > >> 88 ?? ?? ??http://pecl.php.net
>> >> > >> 88 ?? ?? ??http://pecl.php.net/get
>> >> > > These are already mirror bouncers. If you visit the above,
>> >> > > you'll get the closest mirror for downloading.
>> >> > And since there is already ~10 "mirrors" with only one actual
>> >> > backend, should they go to thirdpartymirrors or not ? If not,
>> >> > what about this pseudo-mirrors already present in
>> >> > thirdpartymirrors ?
>> >> I think we should add the pseudo-mirrors, but explicitly mark them
>> >> as such in the file, so that they don't get duplicate entries
>> >> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>> >> Should have just the bouncer).
>> >
>> > It'd be great if we could add some kind of additional mirror
>> > entries, which would be used by repoman to signal missing mirror://
>> > entries but won't be used for downloads.
>>
>> Yep, we could put that in it too:
>> github * * * * * * * *http://github.com/downloads/
>> https://github.com/downloads/
>
> Per spec, portage can choose a random mirror of the list. If we put
> entries like that, these two will be equally possible as the preferred
> cloud. URL -- while they redirect one to another.
>
> We might decide on some common syntax like preceding all extra entries
> with '-' but I don't want to be the one deciding here.

I checked, and current portage code already handle entries starting
with a - gracefully thanks to stack_dictlist (removing them from the
list of mirrors).

--
Corentin Chary
http://xf.iksaif.net
 

Thread Tools




All times are GMT. The time now is 04:39 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org