FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 02-20-2012, 04:03 PM
Corentin Chary
 
Default Gentoo Janitor scripts

Hi,
Since I plan to use the remote remote-id tag for euscan, and I already
use SRC_URI but I'd like all ebuild to use mirrors, I've wrote to
scripts to cleanup your ebuilds and metadata.
There are available here: https://github.com/iksaif/portage-janitor
Here is what you can do with them:

python remoteids.py --diff pycuda Test-Tester Alien-SDL ostinato
--- a/dev-python/pycuda/metadata.xml
+++ b/dev-python/pycuda/metadata.xml
@@ -4,4 +4,7 @@
<maintainer>
<email>spock@gentoo.org</email>
</maintainer>
+ <upstream>
+ <remote-id type="pypi">pycuda</remote-id>
+ </upstream>
</pkgmetadata>
--- a/dev-perl/Alien-SDL/metadata.xml
+++ b/dev-perl/Alien-SDL/metadata.xml
@@ -7,4 +7,7 @@
<email>ssuominen@gentoo.org</email>
<name>Samuli Suominen</name>
</maintainer>
+ <upstream>
+ <remote-id type="cpan">Alien-SDL</remote-id>
+ </upstream>
</pkgmetadata>
--- a/net-analyzer/ostinato/metadata.xml
+++ b/net-analyzer/ostinato/metadata.xml
@@ -7,5 +7,7 @@
</maintainer>
<longdescription lang="en">
</longdescription>
+ <upstream>
+ <remote-id type="google-code">ostinato</remote-id>
+ </upstream>
</pkgmetadata>


$ eix -C dev-python --only-names | python mirrors.py --diff
--- a/dev-python/asciitable/asciitable-0.8.0.ebuild
+++ b/dev-python/asciitable/asciitable-0.8.0.ebuild
@@ -11,7 +11,7 @@

DESCRIPTION="An extensible ASCII table reader"
HOMEPAGE="http://pypi.python.org/pypi/asciitable
http://cxc.harvard.edu/contrib/asciitable"
-SRC_URI="http://pypi.python.org/packages/source/a/${PN}/${P}.tar.gz"
+SRC_URI="mirror://pypi/a/${PN}/${P}.tar.gz"

LICENSE="GPL-2"
SLOT="0"
--- a/dev-python/cosmolopy/cosmolopy-0.1.102.ebuild
+++ b/dev-python/cosmolopy/cosmolopy-0.1.102.ebuild
@@ -15,7 +15,7 @@

DESCRIPTION="Cosmology routines built on NumPy/SciPy"
HOMEPAGE="http://roban.github.com/CosmoloPy/
http://pypi.python.org/pypi/CosmoloPy"
-SRC_URI="http://pypi.python.org/packages/source/C/${MY_PN}/${MY_P}.tar.gz"
+SRC_URI="mirror://pypi/C/${MY_PN}/${MY_P}.tar.gz"

LICENSE="MIT"
SLOT="0"


Feel free to test them, and if they are broken I'll gladly accept a patch .

Maybe some bits could be integrated to repoman...

Thanks,

--
Corentin Chary
http://xf.iksaif.net
 
Old 02-22-2012, 07:48 AM
Corentin Chary
 
Default Gentoo Janitor scripts

I did a quick script to count most used prefixes in SRC_URI yesterday
(https://github.com/iksaif/portage-janitor/blob/master/mirrors.py)

Here is the (filtered) result:

$ eix --only-names | python mirrors.py --count
960 http://dev.gentoo.org
372 http://xorg.freedesktop.org
372 http://xorg.freedesktop.org/releases
372 http://xorg.freedesktop.org/releases/individual
306 http://pear.php.net
306 http://pear.php.net/get
256 http://oss.tresys.com
255 http://oss.tresys.com/files
255 http://oss.tresys.com/files/refpolicy
225 http://hackage.haskell.org/packages
225 http://hackage.haskell.org/packages/archive
225 http://hackage.haskell.org
206 http://ftp.xemacs.org
201 https://github.com
196 http://ftp.xemacs.org/pub
196 http://ftp.xemacs.org/pub/xemacs
193 http://ftp.xemacs.org/pub/xemacs/packages
181 http://gstreamer.freedesktop.org
181 http://gstreamer.freedesktop.org/src
175 http://launchpad.net
175 http://linuxgazette.net
143 http://github.com
130 http://pear.horde.org
130 http://pear.horde.org/get
101 http://savannah.nongnu.org/download
101 http://savannah.nongnu.org
100 http://get.qt.nokia.com
97 ftp://sources.redhat.com/pub
97 ftp://sources.redhat.com
96 http://get.qt.nokia.com/qt
95 http://get.qt.nokia.com/qt/source
90 http://download.gna.org
75 http://pecl.php.net
75 http://pecl.php.net/get
72 http://components.ez.no/get
72 http://components.ez.no
69 https://fedorahosted.org
67 http://www.phrack.org/archives
67 http://www.phrack.org/archives/tgz
67 http://www.phrack.org


From that output we can easilly find out new entries to
thirdpartymirrors, for example:
gentoo-dev http://dev.gentoo.org
xorg http://xorg.freedesktop.org
gna http://download.gna.org
pecl http://pecl.php.net
pear http://pear.php.net
github https://github.com http://github.com
xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
launchpad http://launchpad.net
redhat ftp://sources.redhat.com/pub/ (and probably others !)
etc...

The good part is that once you've modified thirdpartymirrors with new
mirrors, running mirrors.py --all will generate a big patch for all
your ebuilds to use those new mirrors !

--
Corentin Chary
http://xf.iksaif.net
 
Old 02-22-2012, 08:21 AM
Alec Warner
 
Default Gentoo Janitor scripts

On Wed, Feb 22, 2012 at 12:48 AM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> I did a quick script to count most used prefixes in SRC_URI yesterday
> (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py)
>
> Here is the (filtered) result:
>
> $ eix --only-names | python mirrors.py --count
> 960 * * http://dev.gentoo.org
> 372 * * http://xorg.freedesktop.org
> 372 * * http://xorg.freedesktop.org/releases
> 372 * * http://xorg.freedesktop.org/releases/individual
> 306 * * http://pear.php.net
> 306 * * http://pear.php.net/get
> 256 * * http://oss.tresys.com
> 255 * * http://oss.tresys.com/files
> 255 * * http://oss.tresys.com/files/refpolicy
> 225 * * http://hackage.haskell.org/packages
> 225 * * http://hackage.haskell.org/packages/archive
> 225 * * http://hackage.haskell.org
> 206 * * http://ftp.xemacs.org
> 201 * * https://github.com
> 196 * * http://ftp.xemacs.org/pub
> 196 * * http://ftp.xemacs.org/pub/xemacs
> 193 * * http://ftp.xemacs.org/pub/xemacs/packages
> 181 * * http://gstreamer.freedesktop.org
> 181 * * http://gstreamer.freedesktop.org/src
> 175 * * http://launchpad.net
> 175 * * http://linuxgazette.net
> 143 * * http://github.com
> 130 * * http://pear.horde.org
> 130 * * http://pear.horde.org/get
> 101 * * http://savannah.nongnu.org/download
> 101 * * http://savannah.nongnu.org
> 100 * * http://get.qt.nokia.com
> 97 * * *ftp://sources.redhat.com/pub
> 97 * * *ftp://sources.redhat.com
> 96 * * *http://get.qt.nokia.com/qt
> 95 * * *http://get.qt.nokia.com/qt/source
> 90 * * *http://download.gna.org
> 75 * * *http://pecl.php.net
> 75 * * *http://pecl.php.net/get
> 72 * * *http://components.ez.no/get
> 72 * * *http://components.ez.no
> 69 * * *https://fedorahosted.org
> 67 * * *http://www.phrack.org/archives
> 67 * * *http://www.phrack.org/archives/tgz
> 67 * * *http://www.phrack.org
>
>
> From that output we can easilly find out new entries to
> thirdpartymirrors, for example:
> gentoo-dev * *http://dev.gentoo.org

Don't encourage people to do bad things by making it a mirror target

-A

> xorg * * * * * * http://xorg.freedesktop.org
> gna * * * * * * *http://download.gna.org
> pecl * * * * * * http://pecl.php.net
> pear * * * * * * http://pear.php.net
> github * * * * *https://github.com http://github.com
> xemacs * * * http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
> launchpad * *http://launchpad.net
> redhat * * * * ftp://sources.redhat.com/pub/ (and probably others !)
> etc...
>
> The good part is that once you've modified thirdpartymirrors with new
> mirrors, running mirrors.py --all will generate a big patch for all
> your ebuilds to use those new mirrors !
>
> --
> Corentin Chary
> http://xf.iksaif.net
>
 
Old 02-22-2012, 11:20 AM
Mart Raudsepp
 
Default Gentoo Janitor scripts

On K, 2012-02-22 at 09:48 +0100, Corentin Chary wrote:
> I did a quick script to count most used prefixes in SRC_URI yesterday
> (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py)
>
> Here is the (filtered) result:
>
> $ eix --only-names | python mirrors.py --count
> 960 http://dev.gentoo.org
> 372 http://xorg.freedesktop.org
> 372 http://xorg.freedesktop.org/releases
> 372 http://xorg.freedesktop.org/releases/individual
> 306 http://pear.php.net
> 306 http://pear.php.net/get
> 256 http://oss.tresys.com
> 255 http://oss.tresys.com/files
> 255 http://oss.tresys.com/files/refpolicy
> 225 http://hackage.haskell.org/packages
> 225 http://hackage.haskell.org/packages/archive
> 225 http://hackage.haskell.org
> 206 http://ftp.xemacs.org
> 201 https://github.com
> 196 http://ftp.xemacs.org/pub
> 196 http://ftp.xemacs.org/pub/xemacs
> 193 http://ftp.xemacs.org/pub/xemacs/packages
> 181 http://gstreamer.freedesktop.org
> 181 http://gstreamer.freedesktop.org/src
> 175 http://launchpad.net
> 175 http://linuxgazette.net
> 143 http://github.com
> 130 http://pear.horde.org
> 130 http://pear.horde.org/get
> 101 http://savannah.nongnu.org/download
> 101 http://savannah.nongnu.org
> 100 http://get.qt.nokia.com
> 97 ftp://sources.redhat.com/pub
> 97 ftp://sources.redhat.com
> 96 http://get.qt.nokia.com/qt
> 95 http://get.qt.nokia.com/qt/source
> 90 http://download.gna.org
> 75 http://pecl.php.net
> 75 http://pecl.php.net/get
> 72 http://components.ez.no/get
> 72 http://components.ez.no
> 69 https://fedorahosted.org
> 67 http://www.phrack.org/archives
> 67 http://www.phrack.org/archives/tgz
> 67 http://www.phrack.org
>
>
> From that output we can easilly find out new entries to
> thirdpartymirrors, for example:
> gentoo-dev http://dev.gentoo.org
> xorg http://xorg.freedesktop.org
> gna http://download.gna.org
> pecl http://pecl.php.net
> pear http://pear.php.net
> github https://github.com http://github.com
> xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
> launchpad http://launchpad.net
> redhat ftp://sources.redhat.com/pub/ (and probably others !)
> etc...
>
> The good part is that once you've modified thirdpartymirrors with new
> mirrors, running mirrors.py --all will generate a big patch for all
> your ebuilds to use those new mirrors !

If you want this, then you should better figure out actual upstream
mirroring systems and their list of mirrors they would want us to use.
Until such, this seems to be just for shortening SRC_URI addresses when
an upstream tarball domain name or path repeats, and that's definitely
not what thirdpartymirrors is for.


Best,
Mart Raudsepp
 
Old 02-22-2012, 11:36 AM
Corentin Chary
 
Default Gentoo Janitor scripts

On Wed, Feb 22, 2012 at 1:20 PM, Mart Raudsepp <leio@gentoo.org> wrote:
> On K, 2012-02-22 at 09:48 +0100, Corentin Chary wrote:
>> I did a quick script to count most used prefixes in SRC_URI yesterday
>> (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py)
>>
>> Here is the (filtered) result:
>>
>> $ eix --only-names | python mirrors.py --count
>> 960 * * http://dev.gentoo.org
>> 372 * * http://xorg.freedesktop.org
>> 372 * * http://xorg.freedesktop.org/releases
>> 372 * * http://xorg.freedesktop.org/releases/individual
>> 306 * * http://pear.php.net
>> 306 * * http://pear.php.net/get
>> 256 * * http://oss.tresys.com
>> 255 * * http://oss.tresys.com/files
>> 255 * * http://oss.tresys.com/files/refpolicy
>> 225 * * http://hackage.haskell.org/packages
>> 225 * * http://hackage.haskell.org/packages/archive
>> 225 * * http://hackage.haskell.org
>> 206 * * http://ftp.xemacs.org
>> 201 * * https://github.com
>> 196 * * http://ftp.xemacs.org/pub
>> 196 * * http://ftp.xemacs.org/pub/xemacs
>> 193 * * http://ftp.xemacs.org/pub/xemacs/packages
>> 181 * * http://gstreamer.freedesktop.org
>> 181 * * http://gstreamer.freedesktop.org/src
>> 175 * * http://launchpad.net
>> 175 * * http://linuxgazette.net
>> 143 * * http://github.com
>> 130 * * http://pear.horde.org
>> 130 * * http://pear.horde.org/get
>> 101 * * http://savannah.nongnu.org/download
>> 101 * * http://savannah.nongnu.org
>> 100 * * http://get.qt.nokia.com
>> 97 * * *ftp://sources.redhat.com/pub
>> 97 * * *ftp://sources.redhat.com
>> 96 * * *http://get.qt.nokia.com/qt
>> 95 * * *http://get.qt.nokia.com/qt/source
>> 90 * * *http://download.gna.org
>> 75 * * *http://pecl.php.net
>> 75 * * *http://pecl.php.net/get
>> 72 * * *http://components.ez.no/get
>> 72 * * *http://components.ez.no
>> 69 * * *https://fedorahosted.org
>> 67 * * *http://www.phrack.org/archives
>> 67 * * *http://www.phrack.org/archives/tgz
>> 67 * * *http://www.phrack.org
>>
>>
>> From that output we can easilly find out new entries to
>> thirdpartymirrors, for example:
>> gentoo-dev * *http://dev.gentoo.org
>> xorg * * * * * * http://xorg.freedesktop.org
>> gna * * * * * * *http://download.gna.org
>> pecl * * * * * * http://pecl.php.net
>> pear * * * * * * http://pear.php.net
>> github * * * * *https://github.com http://github.com
>> xemacs * * * http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
>> launchpad * *http://launchpad.net
>> redhat * * * * ftp://sources.redhat.com/pub/ (and probably others !)
>> etc...
>>
>> The good part is that once you've modified thirdpartymirrors with new
>> mirrors, running mirrors.py --all will generate a big patch for all
>> your ebuilds to use those new mirrors !
>
> If you want this, then you should better figure out actual upstream
> mirroring systems and their list of mirrors they would want us to use.
> Until such, this seems to be just for shortening SRC_URI addresses when
> an upstream tarball domain name or path repeats, and that's definitely
> not what thirdpartymirrors is for.

Yes, of course, that was just a quick example, not something definitive.

But lets some examples:
- http://xorg.freedesktop.org, it's easy to find mirror for that one,
http://ftp.x.org/pub/ for example
- github: packages seems to use http and https, this script can help
to standardize the url used
- "960 * * http://dev.gentoo.org": that's a lot of package hosted
here, is that really right ?

And still, thirdpartymirrors have some entries with only one "mirror"
and I believe factorizing SRC_URIs is a good thing (if something
changes, you just patch thirdpartymirros, not hundreds of ebuilds).

--
Corentin Chary
http://xf.iksaif.net
 
Old 02-22-2012, 12:33 PM
Ben
 
Default Gentoo Janitor scripts

On 22 February 2012 20:36, Corentin Chary <corentin.chary@gmail.com> wrote:
> - *"960 * * http://dev.gentoo.org": that's a lot of package hosted
> here, is that really right ?

That includes patches >20kb
 
Old 02-22-2012, 04:51 PM
Michał Górny
 
Default Gentoo Janitor scripts

On Wed, 22 Feb 2012 09:48:52 +0100
Corentin Chary <corentin.chary@gmail.com> wrote:

> From that output we can easilly find out new entries to
> thirdpartymirrors, for example:
> gentoo-dev http://dev.gentoo.org

Mirrors shouldn't be used as a dumb URL shortener.


--
Best regards,
Michał Górny
 
Old 02-22-2012, 05:03 PM
Markos Chandras
 
Default Gentoo Janitor scripts

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 02/22/2012 01:33 PM, Ben wrote:
> On 22 February 2012 20:36, Corentin Chary
> <corentin.chary@gmail.com> wrote:
>> - "960 http://dev.gentoo.org": that's a lot of package
>> hosted here, is that really right ?
>
> That includes patches >20kb
>
Gentoo devs are supposed to put patches, tarballs and whatever they
want in their space. This is the recommended policy documented in
devmanual as well

http://devmanual.gentoo.org/general-concepts/mirrors/index.html

- --
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iQIcBAEBCgAGBQJPRS4BAAoJEPqDWhW0r/LCjtsQALww5y0MDZ6zKwx941mPvCgi
QFHqegBv5re+JsqPj/E9IyfzYXG7P06RAp/p5PvaaRM9MRXm5RuBeRCOI+opTDEU
8f0PuQ2Ft3I0+EIdI2V0xcJc5dPQZ9vg/fukApwT/3BOU09VcU5W4MMHjFBS58oX
akeyPTjnKV/TUwD8gUkw/+nnYgDyYjXagoWgJLcTi6EMDYCwu1ax1iRH2kOUTevs
vZHc0Ftejhn8lbemchvfRfqekPR6izhvSE7L1N63SP6cVfY8py kSzELLiqqRSobq
tc7T73uk2gLcWHvmO0aK3YmaJCMjUViiYvMi6lzclyeass7zrK rfTIRuT+1Ffas+
+qSeKpHnIT5yOCEdir8HCoQVoqRjr5pGEMrkLiB91F5I2S9t68 YsvpPP8KhAuXOe
P6aGCE1EpBmcB1LCW3J3MnCJryb0r7fIF+K88ReulDGHn11MJC 8dZGqeBJM224Db
nlN4GYc+nZTt6wOIkv4yyBwhOFLT8szFFHHwt5vBEl6H2SyXJb Ff/nQeiJF/UQkn
tWi88Pa+rk0HeXVzy+2fCg844yYny5PRraqODyfP7LNb0tv0Sc Htx6aJH5Z27emz
Q47OCsdH8v3gAmDDxaYlNu2xVXNoJKJqC9YYVhdI9H08SWq+CY cYl6CC9OrzuqPe
DzMIE+spLzZxJj1+joGt
=FG1X
-----END PGP SIGNATURE-----
 
Old 02-22-2012, 05:11 PM
Corentin Chary
 
Default Gentoo Janitor scripts

On Wed, Feb 22, 2012 at 7:03 PM, Markos Chandras <hwoarang@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 02/22/2012 01:33 PM, Ben wrote:
>> On 22 February 2012 20:36, Corentin Chary
>> <corentin.chary@gmail.com> wrote:
>>> - *"960 * * http://dev.gentoo.org": that's a lot of package
>>> hosted here, is that really right ?
>>
>> That includes patches >20kb
>>
> Gentoo devs are supposed to put patches, tarballs and whatever they
> want in their space. This is the recommended policy documented in
> devmanual as well
>
> http://devmanual.gentoo.org/general-concepts/mirrors/index.html

Yes, you're right, I totally forgot about patches. I'll try to make
the script smarter and skip those.

--
Corentin Chary
http://xf.iksaif.net
 
Old 02-23-2012, 08:49 AM
Corentin Chary
 
Default Gentoo Janitor scripts

> If you want this, then you should better figure out actual upstream
> mirroring systems and their list of mirrors they would want us to use.
> Until such, this seems to be just for shortening SRC_URI addresses when
> an upstream tarball domain name or path repeats, and that's definitely
> not what thirdpartymirrors is for.

Output of mirrors.py --diff --all and then the same output but with
these new mirrors:

xorg http://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/
xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
pear http://pear.php.net/ http://de.pear.php.net/
http://us.pear.php.net/
redhat ftp://sources.redhat.com/pub/
ftp://mirrors.kernel.org/sources.redhat.com/
ftp://mirrors.kernel.org/sources.redhat.com/
http://mirrors.kernel.org/sources.redhat.com/
http://sources-redhat.mirrors.airband.net/
ftp://gd.tuwien.ac.at/gnu/sourceware/
http://gd.tuwien.ac.at/gnu/sourceware/
ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/
http://ftp.gwdg.de/pub/linux/sources.redhat.com/
ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/
http://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/
http://bo.mirror.garr.it/mirrors/sourceware.org/
ftp://bo.mirror.garr.it/mirrors/sourceware.org/
ftp://ftp.mirrorservice.org/sites/sources.redhat.com/pub/
http://www.mirrorservice.org/sites/sources.redhat.com/pub/


fix-mirror.diff: 77 files changed, 79 insertions(+), 79 deletions(-)
fix-mirror-with-new-mirrors: 166 files changed, 291 insertions(+), 291
deletions(-)

Is there a quick way to run `emerge -f` using all mirrors ?

--
Corentin Chary
http://xf.iksaif.net
 

Thread Tools




All times are GMT. The time now is 02:50 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org