Hi,

I recently started working on a small gentoo utility named "euscan"
(for Ebuild Upstream Scan)
For those who don't know debian's uscan, it allows to scan upstream
for new versions. It's used by packages.qa.debian.org (example:
http://packages.qa.debian.org/p/php-net-ipv4.html ).
It's available at: http://xf.iksaif.net/bordel/euscan

Currently, it uses two heuristics to find new versions, both based on
SRC_URI and PV:
- Directory scanning: scan directories to find files with newer version
- Brute Force: generate new possible versions, and try to download files

Note that it also works when only a part of the version is available
in the url.

I think that it would be great to have these informations on
http://packages.gentoo.org/ and/or unofficial
http://gentoo-portage.com/ website.

We could also add the ability to browse packages by maintainer to help
them see if they have any outdated package.

I started this mostly to see if it was possible, and I don't know if
i'll have the time to continue to work on that project, but I think
gentoo really needs an automated way to detect outdated packages.
This could also be a 2011 GSOC project (finishing euscan, and adding a
web based interface to browse the results).

Examples:

$ ./euscan PEAR-Validate-0.8.3
Package: dev-php/PEAR-Validate-0.8.3
Herd: php
Maintainer: none
Location: /usr/portage/dev-php/PEAR-Validate
** Scanning: http://pear.php.net/get/Validate-${PV}.tgz
** Scanning: http://pear.php.net/get
** Generating version from 0.8.3
** Brute forcing: http://pear.php.net/get/Validate-${PV}.tgz
** Trying: http://pear.php.net/get/Validate-0.8.4.tgz ... * * * * * * * [ ok ]
** Trying: http://pear.php.net/get/Validate-0.8.5.tgz ... * * * * * * * [ !! ]
** Trying: http://pear.php.net/get/Validate-0.9.0.tgz ... * * * * * * * [ !! ]
** Trying: http://pear.php.net/get/Validate-0.10.0.tgz ... * * * * * * *[ !! ]
** Trying: http://pear.php.net/get/Validate-0.8.6.tgz ... * * * * * * * [ !! ]
New Upstream Version: 0.8.4 http://pear.php.net/get/Validate-0.8.4.tgz

$ ./euscan icu4j-4.4.2
Package: dev-java/icu4j-4.4.2
Herd: java
Maintainer: none
Location: /usr/portage/dev-java/icu4j
** Scanning: http://download.icu-project.org/files/icu4j/${PV}/icu4j-4_4_2-src.jar
** Scanning: http://download.icu-project.org/files/icu4j
New Upstream Version: 4.5.1 http://download.icu-project.org/files/icu4j/4.5.1/
New Upstream Version: 4.5.2 http://download.icu-project.org/files/icu4j/4.5.2/
New Upstream Version: 4.6 http://download.icu-project.org/files/icu4j/4.6/
New Upstream Version: 4.6rc2 http://download.icu-project.org/files/icu4j/4.6rc2/
New Upstream Version: 4.7.1 http://download.icu-project.org/files/icu4j/4.7.1/
New Upstream Version: 4.6.1 http://download.icu-project.org/files/icu4j/4.6.1/

$ ./euscan IceE-1.3.0-r1
Package: dev-cpp/IceE-1.3.0-r1
Herd: no-herd
Maintainer: maintainer-needed@gentoo.org
Description: The Internet Communications Engine (Ice) is a modern
object-oriented middleware with support for C++, .NET, Java, Python,
Ruby, and PHP
Location: /usr/portage/dev-cpp/IceE
** Scanning: http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz
** Scanning: http://www.zeroc.com/download/IceE
** Generating version from 1.3.0
** Brute forcing:
http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz
** Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.1-linux.tar.gz
... * * *[ !! ]
** Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.2-linux.tar.gz
... * * *[ !! ]
** Trying: http://www.zeroc.com/download/IceE/1.4//IceE-1.4.0-linux.tar.gz
... * * *[ !! ]
** Trying: http://www.zeroc.com/download/IceE/1.5//IceE-1.5.0-linux.tar.gz
... * * *[ !! ]

Thanks,
--
Corentin Chary
http://xf.iksaif.net

Hi,

Corentin Chary <corentin.chary@gmail.com>:
> I started this mostly to see if it was possible, and I don't know if
> i'll have the time to continue to work on that project, but I think
> gentoo really needs an automated way to detect outdated packages.
> This could also be a 2011 GSOC project (finishing euscan, and adding a
> web based interface to browse the results).

Have you applied for a place in GSoC?

V-Li

--
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

On Sat, Mar 26, 2011 at 5:35 PM, Christian Faulhammer <fauli@gentoo.org> wrote:
> Hi,
>
> Corentin Chary <corentin.chary@gmail.com>:
>> I started this mostly to see if it was possible, and I don't know if
>> i'll have the time to continue to work on that project, but I think
>> gentoo really needs an automated way to detect outdated packages.
>> This could also be a 2011 GSOC project (finishing euscan, and adding a
>> web based interface to browse the results).
>
> *Have you applied for a place in GSoC?
>
> V-Li

I can't apply to GSoC, I was suggesting that another student could work on that.

--
Corentin Chary
http://xf.iksaif.net

Hi again,

I found the time to work a little more on euscan:

- Git tree: http://git.iksaif.net/?p=euscan.git
- Demo web interface: http://xf.iksaif.net/dev/.euscan/web/

To feed the web interface, I created a quick&dirty euscan-update. It
currently depends on eix-0.22.5 and gentoolkit-0.3.0.

I launched a full portage scan yesterday, and as you can see on web
interface, it seems to work. You can browse by category, herd, and
maintainer.
The scan is not finished yet (currently scanning net-misc) and keep in
mind that euscan doesn't support mirror:// urls (although, it would be
easy to support some of them).

But there is still a lot to do to make something useful with that, and
I said before, I don't have a lot of time allocated to that project,
so any help would be great.

Quick TODO and ideas:
euscan:
- clean euscan and add options parsing capabilities, with tons of options
- support mirror:// (when possible)
- fix known bugs (see BUGS)

euscan-update:
- clean
- optimize and parallelize
- remove gentoolkit dependency, handle overlays, handle cumulative scans

web interface:
- display all kind of informations (metadatas, keywords, etc...)
- create charts and statistics
- mark/report false positives
- display euscan-update status
- display last euscan output message
- show when euscan can/cannot guess upstream version for a given package

misc:
- automatic bug report
- automatic email report for maintainer/herds

On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:

> misc:
> - automatic bug report
> - automatic email report for maintainer/herds

I'm not sure if this makes sense. For example, it gets dev-lang/ruby
wrong, thinking that our old patch files are missing versions. Also, for
exiftool I only add production releases, so there will almost always be
newer upstream intermediate releases. My point here is that there are
often additional considerations in considering new releases, so fully
automating this is going to be hard. Perhaps a weekly/monthly opt-in
mail to the herd/maintainer with an overview would be useful?

The website is nice to have as another source of scanning for updates,
bookmarked.

Kind regards,

Hans

On Sun, Apr 10, 2011 at 8:43 AM, Hans de Graaff <graaff@gentoo.org> wrote:
> On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:
>
>> misc:
>> - automatic bug report
>> - automatic email report for maintainer/herds
>
> I'm not sure if this makes sense. For example, it gets dev-lang/ruby
> wrong, thinking that our old patch files are missing versions. Also, for
> exiftool I only add production releases, so there will almost always be
> newer upstream intermediate releases. My point here is that there are
> often additional considerations in considering new releases, so fully
> automating this is going to be hard. Perhaps a weekly/monthly opt-in
> mail to the herd/maintainer with an overview would be useful?

Better, what about a per-herd/per-maintainer rss feed ?

> The website is nice to have as another source of scanning for updates,
> bookmarked.

The current site won't be updated, I'm working on a new django-based
website that should be up and running next week.
The url will probably be euscan.iksaif.net, but I'll post it here.

--
Corentin Chary
http://xf.iksaif.net

On Wed, Apr 13, 2011 at 1:47 AM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> On Sun, Apr 10, 2011 at 8:43 AM, Hans de Graaff <graaff@gentoo.org> wrote:
>> On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:
>>
>>> misc:
>>> - automatic bug report
>>> - automatic email report for maintainer/herds
>>
>> I'm not sure if this makes sense. For example, it gets dev-lang/ruby
>> wrong, thinking that our old patch files are missing versions. Also, for
>> exiftool I only add production releases, so there will almost always be
>> newer upstream intermediate releases. My point here is that there are
>> often additional considerations in considering new releases, so fully
>> automating this is going to be hard. Perhaps a weekly/monthly opt-in
>> mail to the herd/maintainer with an overview would be useful?
>
> Better, what about a per-herd/per-maintainer rss feed ?
>
>> The website is nice to have as another source of scanning for updates,
>> bookmarked.
>
> The current site won't be updated, I'm working on a new django-based
> website that should be up and running next week.
> The url will probably be euscan.iksaif.net, but I'll post it here.
>
> --
> Corentin Chary
> http://xf.iksaif.net
>
>

Hi Corentin,

Do you have a public repo for your django code/site available? I would
really enjoy taking a look at what you have here, sounds cool.

Thanks,
Matt
--
Matthew W. Summers

> Hi Corentin,
>
> Do you have a public repo for your django code/site available? I would
> really enjoy taking a look at what you have here, sounds cool.
>
> Thanks,
> Matt

Hi,
Yes, my git repo is here: http://git.iksaif.net/?p=euscan.git;a=summary
Current dependencies are: eix-0.22.5, gentoolkit-0.3.0 and portage of course.

I didn't took the time to write any kind of documentation, but here is
what you'll find in the repo:
- euscan standalone utility
- euscanwww

Currently euscanwww only have basic templates, models and view. But
you can alreay run these commands to fill the database:

$ emerge --sync
$ eix-update
$ python manage.py scan-portage --all
$ python manage.py scan-metadata --all

I'll implement scan-upstream soon.

The old index.php is still available in earlier revisions.

--
Corentin Chary
http://xf.iksaif.net

On Wed, Apr 13, 2011 at 4:58 PM, Corentin Chary
<corentin.chary@gmail.com> wrote:
>> Hi Corentin,
>>
>> Do you have a public repo for your django code/site available? I would
>> really enjoy taking a look at what you have here, sounds cool.
>>
>> Thanks,
>> Matt

Just pushed scan-upstream command (with --parallel support !).
Now, it's time to do a full scan and implement the views.
I staying on #gentoo-unregistered (iksaif) if someone want to get
involved in euscan .

--
Corentin Chary
http://xf.iksaif.net

New website is up and running at

http://euscan.iksaif.net/

The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary

TODO:
- Make some charts to see how it's going
- Finish the "scan my world" feature
- Add a way to subsribe to herds/maintainer/packages in order to
receive weekly/monthly reports

I'll gladly accept any patch !

--
Corentin Chary
http://xf.iksaif.net