FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 09-30-2012, 02:01 PM
Michał Górny
 
Default Initial python-r1.eclass & distutils-r1.eclass

On Sat, 29 Sep 2012 22:48:00 +0200
hasufell <hasufell@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 09/29/2012 08:39 PM, Michał Górny wrote:
> > On Sat, 29 Sep 2012 16:37:15 +0200 Dirkjan Ochtman <djc@gentoo.org>
> > wrote:
> >
> >> On Sat, Sep 29, 2012 at 4:26 PM, hasufell <hasufell@gentoo.org>
> >> wrote:
> >>> That still does not explain the reasons why this work was
> >>> initiated.
> >>>
> >>> If there is any way to fix the current eclass, that should be
> >>> preferred.
> >>
> >> I tend to agree. Michał, let me first say I value the time you
> >> have invested to make the eclasses better. However, at this point
> >> I have a strong feeling that we have more people willing to write
> >> code to fix things than we have people building consensus on
> >> what features/policies/mechanisms we need to make it easy to
> >> write high-quality ebuilds for Python/distutils. I would prefer
> >> discussions on problems that the current ebuilds have and
> >> discussions on how to solve them, not at the code level, but that
> >> the mechanism level.
> >
> > The main issue: noone wants to even touch python.eclass or
> > anything nearby.
> >
> > The second issue: python-distutils-ng isn't good enough. It has
> > too many things hard-wired. I think I have already pointed enough
> > problems with it. Not that many people cared to respond.
> >
> > It's sad that people don't care to respond when you point the
> > issues out but then complain when you do something to fix them.
> >
>
> Did you CC gentoo-dev? I cannot find the tread.

Maybe. People interested in Python should be either on the Python ml
or on the python@ alias, I believe.

> > [example needed]
> >
>
> ??
>
> I meant that not all tree ebuilds use the same python-eclass
> implementation which IS a problem. Adding another implementation does
> not really improve that situation.

As long as they can inter-operate correctly, I don't see any problem.

> > Please list the features. Preferably, order them by usefulness,
> > with exact use cases.
> >
>
> As I said, I think the python herd already did a list on this.
>
> Btw. could you give exact examples on how to convert widely used
> python ebuilds with your eclasses?
> E.g. dev-python/pygobject dev-python/setuptools or dev-libs/boost?

I have prepared and tested an ebuild for pygobject and setuptools here:

http://git.overlays.gentoo.org/gitweb/?p=dev/mgorny.git;a=tree;f=dev-python;h=557b96c041bb6da969574dd75dcdfd4022ba636b; hb=refs/heads/python-r1

I will take a look at boost a while later, since I have to have much
more time to compile it :P.

> How can I convert shebangs consistently and recursively?

Converting shebangs applies to packages supporting multiple Python
implementations or those not doing so?

--
Best regards,
Michał Górny
 
Old 09-30-2012, 02:12 PM
Fabian Groffen
 
Default Initial python-r1.eclass & distutils-r1.eclass

I'm on the list, obviously, so PLEASE stop Cc-ing me!

On 30-09-2012 15:57:16 +0200, Michał Górny wrote:
> > The files are indeed cache, and should be generated on the system that
> > installs the files, not the system that builds them. They are currently
> > outside of VDB. pyc files store the path to the original files, so
> > generating in ${ROOT} yields in wrong paths. Python sometimes
> > regenerates the files when it runs. It may as well just ignore them
> > (since they are newer but non-matching, unclear to me). In the worst
> > case it returns "Bad marshalling data".
>
> I'm afraid you are wrong here.
>
> I've just tested an ebuild using distutils (flaggie) and autotools
> (pygobject) and both install .pyc files with correct paths (i.e. with
> DESTDIR stripped).

Could be, if it uses relative paths, but they won't be absolute then
either. From some experiments I did, I saw Python ignoring wrong paths
(entire cache?), or updating the pyc/pyo files. Regardless whether or
not the data is wrong, the discussion is more if it hurts (and makes
sense).

Back then, we found it hurt us (Bad marshalling data) when using
binpkgs. Not sure if it still does today with Python 2.7. Somehow we
reached a consensus with the Python maintainer at that time that cache
stuff shouldn't be in VDB, also because it depends on system and perhaps
Python version. Since embedded systems might not even want it (it's
just cache afterall, wasting their sometimes precious disk space), it
had to be out of the package contents for them as well.

That's how we came to the situation as it was before python eclass
rewrites. I'd much prefer Python to write its cache to some
/var/cache/python dir so it can create whatever it wants, but on a
place which is not of any concern, as its obvious one can throw it away,
and doesn't pollute the live fs. But Python doesn't work that way.

Whatever the new way will be, for whatever reason, please make sure it
is consistent. And be aware that this doesn't just mean changing
eclasses, since this rule is effectuated in certain ebuilds too
(dev-lang/python itself being the most obvious example).


--
Fabian Groffen
Gentoo on a different level
 
Old 09-30-2012, 06:15 PM
Zac Medico
 
Default Initial python-r1.eclass & distutils-r1.eclass

On 09/30/2012 07:12 AM, Fabian Groffen wrote:
> Back then, we found it hurt us (Bad marshalling data) when using
> binpkgs. Not sure if it still does today with Python 2.7. Somehow we
> reached a consensus with the Python maintainer at that time that cache
> stuff shouldn't be in VDB, also because it depends on system and perhaps
> Python version. Since embedded systems might not even want it (it's
> just cache afterall, wasting their sometimes precious disk space), it
> had to be out of the package contents for them as well.

They can use INSTALL_MASK="*.py[co]" if they need to. In portage we have
a default COLLISION_IGNORE="*.py[co]" setting, so that there are no
related issues with file collisions for *.py[co] if the user removes the
INSTALL_MASK setting later.
--
Thanks,
Zac
 
Old 09-30-2012, 09:47 PM
Brian Harring
 
Default Initial python-r1.eclass & distutils-r1.eclass

On Sun, Sep 30, 2012 at 10:58:06AM +0200, Fabian Groffen wrote:
> On 30-09-2012 10:31:17 +0200, Pacho Ramos wrote:
> > > > Personally, I usually run:
> > > > - python_clean_py-compile_files -> Clean py-compile files to disable
> > > > byte-compilation allowing us to drop all various ways of doing this that
> > > > were living in the tree some time ago.
> > >
> > > Hmm, what's the problem with compiling them? Do you mean some case when
> > > the results of the compilation are different from the way done
> > > by the eclass?
> > >
> >
> > Well, if I don't misremember, we currently prefer to compile them at
> > postinst phase instead of during src_compile, but maybe this is no
> > longer needed (no idea )
>
> The files are indeed cache, and should be generated on the system that
> installs the files, not the system that builds them. They are currently
> outside of VDB. pyc files store the path to the original files, so
> generating in ${ROOT} yields in wrong paths. Python sometimes
> regenerates the files when it runs. It may as well just ignore them
> (since they are newer but non-matching, unclear to me).

.pyc should be compatible within slotted python versions;
recompilation occurs if .pyc/.pyo mtime differs from the originating
.py. Via EAPI=3 mtime gurantees, that should be addressed- below
that, compilation via pkg_postinst is necessary due to the lack of
mtime guarantees.

> In the worst case it returns "Bad marshalling data".

Examples wanted for this. If this occurs, that's a python bug- one
exception... portage (figures). They install into a non
/usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is
exposed/accessed for py2.7 for example.

That said, a .pyc/.pyo from an older python version causing a blow up
in a differing python version hasn't occurred since the 2.3 -> 2.4
transition (if memory serves).

Either way, examples desired please.
~harring
 
Old 10-01-2012, 06:36 AM
Fabian Groffen
 
Default Initial python-r1.eclass & distutils-r1.eclass

On 30-09-2012 14:47:17 -0700, Brian Harring wrote:
> > In the worst case it returns "Bad marshalling data".
>
> Examples wanted for this. If this occurs, that's a python bug- one
> exception... portage (figures). They install into a non
> /usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is
> exposed/accessed for py2.7 for example.

https://bugs.gentoo.org/show_bug.cgi?id=300922

I doubt whether it's a Python bug, we have to mess with the files. But
then again, I did some toying, and it seems Python doesn't care about
this (any more?).


--
Fabian Groffen
Gentoo on a different level
 
Old 10-01-2012, 07:19 AM
Brian Harring
 
Default Initial python-r1.eclass & distutils-r1.eclass

On Mon, Oct 01, 2012 at 08:36:12AM +0200, Fabian Groffen wrote:
> On 30-09-2012 14:47:17 -0700, Brian Harring wrote:
> > > In the worst case it returns "Bad marshalling data".
> >
> > Examples wanted for this. If this occurs, that's a python bug- one
> > exception... portage (figures). They install into a non
> > /usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is
> > exposed/accessed for py2.7 for example.
>
> https://bugs.gentoo.org/show_bug.cgi?id=300922
>
> I doubt whether it's a Python bug, we have to mess with the files. But
> then again, I did some toying, and it seems Python doesn't care about
> this (any more?).

Well, offhand that bug is pre EAPI3 (eapi3 was approved 01/18/10, and
adoption was slow- lot of people skipped straight to eapi4) - so the
mtime wouldn't have been guaranteed preserved for a long while.
Meaning the bugs data I don't trust to be relevant due to timing, and
age.

As you said, this needs revisiting- minimally, portage is screwing
around contents there, and I don't trust the python eclass to /not/ be
forcing a compileall after the fact anyways.

Suggest backing down the various protections for a full test, and
resuming that bug- if you can replicate it, I'm definitely interested
(dealt with this when it occurred for 2.3->2.4 for example).

~harring
 
Old 10-01-2012, 03:17 PM
Marien Zwart
 
Default Initial python-r1.eclass & distutils-r1.eclass

(pardon any awkward formatting, writing from webinterface instead of a
"proper" client. Please yell at me off-list or on irc if this post is
excessively broken.)

> https://bugs.gentoo.org/show_bug.cgi?id=300922

From that bug:

> - chpathtool does some "simple" heuristic to switch the build EPREFIX
> to the new target EPREFIX
>
> - python (seemingly) doesn't like when the pyc files are changed and *not* regenerated.
> You will see the message: "Bad marshalling data"

I'm not sure if I'm interpreting this correctly, but it sounds as if
your chpathtool edits the .pyc file to replace the path? If yes: that
will indeed not work, as strings are stored length-prefixed. You
cannot sensibly edit .pyc files, you should normally just rewrite
them.

The .pyc/.pyo file store a version-dependent magic, the mtime of their
corresponding .py file, and (since recently) the size of the
correspondig .py file. If either changes the .pyc/.pyo is regenerated.
The magic does not normally change in released pythons, so that's not
much of a problem. The mtime is probably no longer a problem these
days either.

There used to be a more subtle problem: the python objects you import
have the path to their source embedded into them, and if you move the
.pyc/.pyo file around that path would be wrong, confusing some (mostly
debugging) tools (see http://bugs.python.org/issue1180193 ). This was
fixed fairly recently (in python 3, python 2.7 and python 2.6.2), so
it is probably no longer worth worrying about, but it's probably one
reason these files were handled this way in the past.

--
Marien Zwart (marienz)
 
Old 10-01-2012, 03:44 PM
Fabian Groffen
 
Default Initial python-r1.eclass & distutils-r1.eclass

On 01-10-2012 17:17:40 +0200, Marien Zwart wrote:
> There used to be a more subtle problem: the python objects you import
> have the path to their source embedded into them, and if you move the
> .pyc/.pyo file around that path would be wrong, confusing some (mostly
> debugging) tools (see http://bugs.python.org/issue1180193 ). This was
> fixed fairly recently (in python 3, python 2.7 and python 2.6.2), so
> it is probably no longer worth worrying about, but it's probably one
> reason these files were handled this way in the past.

This most likely is the reason why my tests with Python 2.7 worked fine
when I tinkered with the .pyc and/or moved it around.

Thanks

--
Fabian Groffen
Gentoo on a different level
 
Old 10-05-2012, 12:42 PM
Michał Górny
 
Default Initial python-r1.eclass & distutils-r1.eclass

On Sat, 29 Sep 2012 10:53:29 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Instead of the floating patches and p-d-ng modifications I sent
> earlier, here are the two complete (so far, well, initial :P) eclasses
> for review.
>
> They are designed as 'mostly' drop-in python-distutils-ng replacement.

Conversion of all ebuilds in my overlay:
http://git.overlays.gentoo.org/gitweb/?p=dev/mgorny.git;a=commitdiff;h=c0c8c4ea

Conversion of all ebuilds in gx86:
https://bitbucket.org/mgorny/gx86-working-tree/changeset/6442b5d5b

--
Best regards,
Michał Górny
 

Thread Tools




All times are GMT. The time now is 07:43 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org