First steps for creating an API for portage
 

Ok guys :)

I think it has been consensus, that we need to define the operations
everyone wants to be in the API.

The following is currently only my point of view, but I'll represent it
here also:

Then a python-API is created which should allow for this set of
operations. From this one could go and implement the library to make the
API do something useful :)

In parallel (or thereafter), we build the C-bindings. The API for these
bindings probably look different -- but I guess they should be
implemented in terms of the library created above.

By example:

- Operation: get the list of categories
- Python-API: portage.api.categories() : Category list
- Implementation: def categories(): return ....
- C-API: category * categories()
- C-Implementation: some wrapper around portage.api.categories


So ... first step: Which operations do we need?

- René

First steps for creating an API for portage
 

On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote:
> In parallel (or thereafter), we build the C-bindings. The API for these
> bindings probably look different -- but I guess they should be
> implemented in terms of the library created above.
>
> By example:
>
> - Operation: get the list of categories
> - Python-API: portage.api.categories() : Category list
> - Implementation: def categories(): return ....
> - C-API: category * categories()
> - C-Implementation: some wrapper around portage.api.categories

If you want to deliver a C implementation, I'd wonder why you wouldn't
just make a full C implementation and create Python wrappers for them
instead of the other way around. Might accidentially speed up Portage,
and make tools like portage-utils happy.


--
Fabian Groffen
Gentoo on a different level

First steps for creating an API for portage
 

On Fri, Jun 18, 2010 at 08:35:13AM +0200, Fabian Groffen wrote:
> On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote:
> > In parallel (or thereafter), we build the C-bindings. The API for these
> > bindings probably look different -- but I guess they should be
> > implemented in terms of the library created above.
> >
> > By example:
> >
> > - Operation: get the list of categories
> > - Python-API: portage.api.categories() : Category list
> > - Implementation: def categories(): return ....
> > - C-API: category * categories()
> > - C-Implementation: some wrapper around portage.api.categories
>
> If you want to deliver a C implementation, I'd wonder why you wouldn't
> just make a full C implementation and create Python wrappers for them
> instead of the other way around. Might accidentially speed up Portage,
> and make tools like portage-utils happy.

Got a couple of years of developer time you can spare? ;)

Portage's slowness ain't python, if in doubt look at pkgcore. Even
w/out our extensions we blow it's lid off. So kindly skip the "it'll
be faster in c" bits... that joke is beyond dead at this point ;)

back to the subject at hand...

While I'm not generally a fan of embedding python, in this case it's
what makes sense. That said I'm not hugely convinced the proposal on
the table is accurate- knocking out a public portage API needs to
occur, but a c-api is a very large and seperate beast from a public
python API- namely due to crossing the vm, having to do FFI
conversions, etc. It's not the simplest thing.

For pkgkit, what I'd suggest is basically pushing into pkgkit just cpy
stubs that know to import a specific namespace, and grab functions
from there. It's been a while since I've gone through that code, but
if memory serves this shouldn't be too hard to do- the reasons for
doing this pretty much come down to allowing portage to minimize the
area it needs to provide a stable API for.

If all it has to provide is a stable set of functors for pkgkit,
that leaves open fixing portage internals/architecture.

My two cents either way. I know for pkgcore/pkgkit, this is my
intention also...

~harring

First steps for creating an API for portage
 

On Fri, 2010-06-18 at 08:35 +0200, Fabian Groffen wrote:
> On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote:
> > In parallel (or thereafter), we build the C-bindings. The API for these
> > bindings probably look different -- but I guess they should be
> > implemented in terms of the library created above.
> >
> > By example:
> >
> > - Operation: get the list of categories
> > - Python-API: portage.api.categories() : Category list
> > - Implementation: def categories(): return ....
> > - C-API: category * categories()
> > - C-Implementation: some wrapper around portage.api.categories
>
> If you want to deliver a C implementation, I'd wonder why you wouldn't
> just make a full C implementation and create Python wrappers for them
> instead of the other way around. Might accidentially speed up Portage,
> and make tools like portage-utils happy.
>
>

Unless I'm way off the mark, your suggesting to re-code portage into "C"
then create a python interface to it. Others have started such a
project, but none have completed aside from paludis (C++).

What we are trying to do is make a higher level interface API that will
do several low level portage operations/calls to produce data for API
consumers like the gui frontends to portage. To do that in "C" would
require several data conversions from python data types to CTYPES and
back. Then convert the final results back to python for python consumer
APPS. I do not see how that would speed up portage as emerge and normal
portage operations will not have anything to do with this API aside from
the consumer apps calls to this api that will in turn run some emerge
operations or do data gathering via portage function calls.

Currently there are 3 gui's that are python based (porthole, portato and
kportagetray), 3 that are in other languages (Himerge in Haskel, Kuroo
in C++, packagekit in C). These last 3 can make use of a "C" interface
to this new API instead of trying to decode the metadata cache directly,
etc..

If there are other apps that would like to get info from portage via
this API, that is fine with me. I have intentions of making the "C"
interface available to all apps that desire to. If there is data that
they want to get from portage via the API, then they should send their
wish list in and we'll do our best to get it in there for them.

Currently the API will be based on the packagekit portage backend code
that was produced last summer and other code from porthole, portato,
kportagetray. We are also going to put together a layman API for
consumer apps (guis frontends) to use to operate layman without the need
to run it in and parse terminal output. It too will hopefully be
available via a "C" interface for non-python apps.
--
Brian Dolbec <brian.dolbec@gmail.com>

First steps for creating an API for portage
 

On 18-06-2010 01:27:57 -0700, Brian Dolbec wrote:
> Unless I'm way off the mark, your suggesting to re-code portage into "C"
> then create a python interface to it. Others have started such a
> project, but none have completed aside from paludis (C++).

It's actually on my wishlist for a long time, just because Python is an
extreme pain in my environments, and given its current maintainer it is
going to be an extreme pain for a long time as well.

> If there are other apps that would like to get info from portage via
> this API, that is fine with me. I have intentions of making the "C"
> interface available to all apps that desire to. If there is data that
> they want to get from portage via the API, then they should send their
> wish list in and we'll do our best to get it in there for them.
>
> Currently the API will be based on the packagekit portage backend code
> that was produced last summer and other code from porthole, portato,
> kportagetray. We are also going to put together a layman API for
> consumer apps (guis frontends) to use to operate layman without the need
> to run it in and parse terminal output. It too will hopefully be
> available via a "C" interface for non-python apps.

I don't want to push you into the C corner, but tools like qfile and
qlist are in orders of magnitude faster than their equery variants for a
good reason. Maybe this simple command gives you an idea why I think
python and portage are slow.

% time portageq envvar CHOST
powerpc-apple-darwin8
0.488u 0.704s 0:03.09 38.1% 0+0k 25+70io 0pf+0w

(three full seconds to return a very simple var)

That said, if you design C APIs, please design them from a C point of
view, initially implemented by your Python functionality doing the
necessary wrapping to get a sane C structure. Then they can be replaced
by native C code as RSI and time permits in the future.


--
Fabian Groffen
Gentoo on a different level

First steps for creating an API for portage
 

Am 18.06.2010 09:55, schrieb Brian Harring:
> While I'm not generally a fan of embedding python, in this case it's
> what makes sense. That said I'm not hugely convinced the proposal on
> the table is accurate- knocking out a public portage API needs to
> occur, but a c-api is a very large and seperate beast from a public
> python API- namely due to crossing the vm, having to do FFI
> conversions, etc. It's not the simplest thing.

Well - there a tools like Cython which will take large parts of this
burden from your shoulder.

> For pkgkit, what I'd suggest is basically pushing into pkgkit just cpy
> stubs that know to import a specific namespace, and grab functions
> from there. It's been a while since I've gone through that code, but
> if memory serves this shouldn't be too hard to do- the reasons for
> doing this pretty much come down to allowing portage to minimize the
> area it needs to provide a stable API for.
>
> If all it has to provide is a stable set of functors for pkgkit,
> that leaves open fixing portage internals/architecture.

I thought this was the whole idea of having such an API - providing a
stable set of higher level functions, whose implementation can change if
needed.

Just pulling random portage functions into a new namespace is not a goal
imho.

- René

First steps for creating an API for portage
 

Am 18.06.2010 11:20, schrieb Fabian Groffen:
> That said, if you design C APIs, please design them from a C point of
> view, initially implemented by your Python functionality doing the
> necessary wrapping to get a sane C structure. Then they can be replaced
> by native C code as RSI and time permits in the future.

This has been the idea. Perhaps not made clearly enough in my initial
mail :)

- René

First steps for creating an API for portage
 

On Fri, Jun 18, 2010 at 12:55:37PM +0200, Rennn 'Necoro' Neumann wrote:
> Am 18.06.2010 09:55, schrieb Brian Harring:
> > While I'm not generally a fan of embedding python, in this case it's
> > what makes sense. That said I'm not hugely convinced the proposal on
> > the table is accurate- knocking out a public portage API needs to
> > occur, but a c-api is a very large and seperate beast from a public
> > python API- namely due to crossing the vm, having to do FFI
> > conversions, etc. It's not the simplest thing.
>
> Well - there a tools like Cython which will take large parts of this
> burden from your shoulder.

Cython is for extending python, embedding python. Cython doesn't do
jack for embedding... frankly very little does.


> > For pkgkit, what I'd suggest is basically pushing into pkgkit just cpy
> > stubs that know to import a specific namespace, and grab functions
> > from there. It's been a while since I've gone through that code, but
> > if memory serves this shouldn't be too hard to do- the reasons for
> > doing this pretty much come down to allowing portage to minimize the
> > area it needs to provide a stable API for.
> >
> > If all it has to provide is a stable set of functors for pkgkit,
> > that leaves open fixing portage internals/architecture.
>
> I thought this was the whole idea of having such an API - providing a
> stable set of higher level functions, whose implementation can change if
> needed.
>
> Just pulling random portage functions into a new namespace is not a goal
> imho.

Everything I've seen of the pkgkit hooks required for this, it's not a
good api. It's a good API for *pkgkit* which targets LCD, but that
doesn't mean it's a good *portage* api to expose and maintain.

As such a portage provided/bundled shim is the best notion, at least
until portage locks down a stable api it's willing to support rather
than "sure, what we've got we'll call stable!".

~harring