FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 12-19-2009, 02:24 PM
Arfrever Frehtes Taifersar Arahesis
 
Default Versioning of Python scripts

Distutils/Setuptools/Distribute modify shebangs of installed Python scripts, so that they
contain path of Python interpreter with version included (e.g. "#!/usr/bin/python3.2").
This behavior has both advantage and disadvantages:
- Scripts of packages supporting only e.g. Python 2 can be executed (without necessity
of using of e.g. "python2 /usr/bin/${script}") after activating of e.g. Python 3.
- Scripts of packages supporting multiple Python versions ignore active Python version.
- Scripts of packages supporting multiple Python versions cannot be easily (without
necessity of using of e.g. "python3.1 /usr/bin/${script}") executed with a Python
version different than active Python version.
The best solution, which removes these 2 disadvantages and preserves the advantage, seems
to be to rename Python scripts to include Python version [1] in filenames, and create wrapper
scripts, which call appropriate target scripts [2]. Some files sometimes try to execute
e.g. "/usr/bin/python /usr/bin/${script}", so wrapper scripts must be implemented in Python.
Wrapper scripts try to execute "${wrapper_script}-${PYTHON_ABI}" files (e.g. "py.test" will
execute "py.test-3.1", when Python 3.1 is set as active Python version).

distutils.eclass will automatically rename some scripts [3] in "${D}usr/bin" and call
the function, which generates wrapper scripts. In case somebody is interested in reading of
source code of python_generate_wrapper_scripts() function and potential suggesting of
improvements, I'm attaching this function and 2 example wrapper scripts. I'm planning to
commit addition of this function in next week.

[1] Actually Python ABI.
[2] Wrapper scripts can also be created for ELF target files (e.g. "mpipython").
[3] Only scripts of packages supporting installation for multiple Python versions will be
renamed. Scripts, whose filenames end in [[:digit:]]+.[[:digit:]]+, won't be renamed.

--
Arfrever Frehtes Taifersar Arahesis
# @FUNCTION: python_generate_wrapper_scripts
# @USAGE: [-E|--respect-EPYTHON] [-f|--force] [-q|--quiet] [--] <file> [files]
# @DESCRIPTION:
# Generate wrapper scripts. Existing files are overwritten only with --force option.
# If --respect-EPYTHON option is specified, then generated wrapper scripts will
# respect EPYTHON variable at run time.
python_generate_wrapper_scripts() {
local eselect_python_option file force="0" quiet="0" PYTHON_ABI python2_enabled="0" python2_supported_versions python3_enabled="0" python3_supported_versions respect_EPYTHON="0"
python2_supported_versions="2.4 2.5 2.6 2.7"
python3_supported_versions="3.0 3.1 3.2"

while (($#)); do
case "$1" in
-E|--respect-EPYTHON)
respect_EPYTHON="1"
;;
-f|--force)
force="1"
;;
-q|--quiet)
quiet="1"
;;
--)
break
;;
-*)
die "${FUNCNAME}(): Unrecognized option '$1'"
;;
*)
break
;;
esac
shift
done

validate_PYTHON_ABIS
for PYTHON_ABI in ${python2_supported_versions}; do
if has "${PYTHON_ABI}" ${PYTHON_ABIS}; then
python2_enabled="1"
fi
done
for PYTHON_ABI in ${python3_supported_versions}; do
if has "${PYTHON_ABI}" ${PYTHON_ABIS}; then
python3_enabled="1"
fi
done

if [[ "${python2_enabled}" == "1" && "${python3_enabled}" == "1" ]]; then
eselect_python_option=
elif [[ "${python2_enabled}" == "1" && "${python3_enabled}" == "0" ]]; then
eselect_python_option="--python2"
elif [[ "${python2_enabled}" == "0" && "${python3_enabled}" == "1" ]]; then
eselect_python_option="--python3"
else
die "${FUNCNAME}(): Unsupported environment"
fi

for file in "$@"; do
if [[ -f "${file}" && "${force}" == "0" ]]; then
die "${FUNCNAME}(): '$1' already exists"
fi

if [[ "${quiet}" == "0" ]]; then
einfo "Generating '${file#${D%/}}' wrapper script"
fi

cat << EOF > "${file}"
#!/usr/bin/env python
# Gentoo '${file##*/}' wrapper script

import os
import re
import subprocess
import sys

EPYTHON_re = re.compile(r"^python(d+.d+)$")

EOF
if [[ "$?" != "0" ]]; then
die "${FUNCNAME}(): Generation of '$1' failed"
fi
if [[ "${respect_EPYTHON}" == "1" ]]; then
cat << EOF >> "${file}"
EPYTHON = os.environ.get("EPYTHON")
if EPYTHON:
EPYTHON_matched = EPYTHON_re.match(EPYTHON)
if EPYTHON_matched:
PYTHON_ABI = EPYTHON_matched.group(1)
else:
sys.stderr.write("EPYTHON variable has unrecognized value '%s'
" % EPYTHON)
sys.exit(1)
else:
try:
eselect_process = subprocess.Popen(["/usr/bin/eselect", "python", "show"${eselect_python_option:+, $(echo """)}${eselect_python_option}${eselect_python_opti on:+$(echo """)}], stdout=subprocess.PIPE)
if eselect_process.wait() != 0:
raise ValueError
except (OSError, ValueError):
sys.stderr.write("Execution of 'eselect python show${eselect_python_option:+ }${eselect_python_option}' failed
")
sys.exit(1)

eselect_output = eselect_process.stdout.read()
if not isinstance(eselect_output, str):
# Python 3
eselect_output = eselect_output.decode()

EPYTHON_matched = EPYTHON_re.match(eselect_output)
if EPYTHON_matched:
PYTHON_ABI = EPYTHON_matched.group(1)
else:
sys.stderr.write("'eselect python show${eselect_python_option:+ }${eselect_python_option}' printed unrecognized value '%s" % eselect_output)
sys.exit(1)
EOF
if [[ "$?" != "0" ]]; then
die "${FUNCNAME}(): Generation of '$1' failed"
fi
else
cat << EOF >> "${file}"
try:
eselect_process = subprocess.Popen(["/usr/bin/eselect", "python", "show"${eselect_python_option:+, $(echo """)}${eselect_python_option}${eselect_python_opti on:+$(echo """)}], stdout=subprocess.PIPE)
if eselect_process.wait() != 0:
raise ValueError
except (OSError, ValueError):
sys.stderr.write("Execution of 'eselect python show${eselect_python_option:+ }${eselect_python_option}' failed
")
sys.exit(1)

eselect_output = eselect_process.stdout.read()
if not isinstance(eselect_output, str):
# Python 3
eselect_output = eselect_output.decode()

EPYTHON_matched = EPYTHON_re.match(eselect_output)
if EPYTHON_matched:
PYTHON_ABI = EPYTHON_matched.group(1)
else:
sys.stderr.write("'eselect python show${eselect_python_option:+ }${eselect_python_option}' printed unrecognized value '%s" % eselect_output)
sys.exit(1)
EOF
if [[ "$?" != "0" ]]; then
die "${FUNCNAME}(): Generation of '$1' failed"
fi
fi
cat << EOF >> "${file}"

target_executable = "%s-%s" % (sys.argv[0], PYTHON_ABI)
if not os.path.exists(target_executable):
sys.stderr.write("'%s' does not exist
" % target_executable)
sys.exit(1)

os.execv(target_executable, sys.argv)
EOF
if [[ "$?" != "0" ]]; then
die "${FUNCNAME}(): Generation of '$1' failed"
fi
fperms +x "${file#${D%/}}" || die "fperms '${file}' failed"
done
}
 
Old 12-21-2009, 09:50 AM
Brian Harring
 
Default Versioning of Python scripts

On Sat, Dec 19, 2009 at 04:24:49PM +0100, Arfrever Frehtes Taifersar Arahesis wrote:
> Distutils/Setuptools/Distribute modify shebangs of installed Python scripts, so that they
> contain path of Python interpreter with version included (e.g. "#!/usr/bin/python3.2").
> This behavior has both advantage and disadvantages:
> - Scripts of packages supporting only e.g. Python 2 can be executed (without necessity
> of using of e.g. "python2 /usr/bin/${script}") after activating of e.g. Python 3.
> - Scripts of packages supporting multiple Python versions ignore active Python version.
> - Scripts of packages supporting multiple Python versions cannot be easily (without
> necessity of using of e.g. "python3.1 /usr/bin/${script}") executed with a Python
> version different than active Python version.
> The best solution, which removes these 2 disadvantages and preserves the advantage, seems
> to be to rename Python scripts to include Python version [1] in filenames, and create wrapper
> scripts, which call appropriate target scripts [2]. Some files sometimes try to execute
> e.g. "/usr/bin/python /usr/bin/${script}", so wrapper scripts must be implemented in Python.
> Wrapper scripts try to execute "${wrapper_script}-${PYTHON_ABI}" files (e.g. "py.test" will
> execute "py.test-3.1", when Python 3.1 is set as active Python version).
>
> distutils.eclass will automatically rename some scripts [3] in "${D}usr/bin" and call
> the function, which generates wrapper scripts. In case somebody is interested in reading of
> source code of python_generate_wrapper_scripts() function and potential suggesting of
> improvements, I'm attaching this function and 2 example wrapper scripts. I'm planning to
> commit addition of this function in next week.

Not really a huge fan of the EPYTHON var... can you clarify it's real
world usage? I can see that causing all sorts of mayhem as it passes
it's way down through python scripts invoking other scripts-
specifically thinking of a py3k only script being forced to 3.1, then
invoking a py2k script.

Beyond that, please provide a way to *disable* this for a pkg. At
least for pkgcore, I've already been looking at ways to deal with
this and would rather solve it at the pkg level (since EPYTHON let
alone eselect may not exist for certain target deployments of
pkgcore itself).

Basically, no point in having wrapper scripts if the target already
can do it's own version of this, hence wanting a way to disable it in
the ebuild- that's just for the script mangling, library installing
for multiple python abis is a seperate thing.

Aside from that, punting on the re import might be nice primarily for
speed reasons (no it's not a huge import in cost, but this is an extra
~.025 per python script invoked, ignoring the ~.3 to ~.02 for eselect
dependant on cache status).

~harring
 
Old 12-22-2009, 08:04 AM
Peter Volkov
 
Default Versioning of Python scripts

В Сбт, 19/12/2009 в 16:24 +0100, Arfrever Frehtes Taifersar Arahesis
пишет:
> - Scripts of packages supporting only e.g. Python 2 can be executed (without necessity
> of using of e.g. "python2 /usr/bin/${script}") after activating of e.g. Python 3.
> - Scripts of packages supporting multiple Python versions ignore active Python version.

> The best solution, which removes these 2 disadvantages and preserves the advantage, seems
> to be to rename Python scripts to include Python version [1] in filenames, and create wrapper
> scripts, which call appropriate target scripts [2].

But still additional wrapper scripts and increased number of scripts in
total is not really nice. What do you think about another solution:

Separate possible script's python API versions into database, e.g.
in /var/db/pyabi/${CATEGORY}/${PN}-${SLOT}. Create python wrapper only
for unmerged but still in use (found from pyabi db) python versions,
e.g. /usr/bin/python3.2 in case you've unmerged python, but scripts
still reference python3.2.

This way scripts that use python versions that are still in the tree
will work without any overhead, fewer scripts in /{usr,}bin and during
python merge/unmerge it is possible to warn user about scripts that use
unmerged python versions (and thus run with some overhead). I guess
still we can add some user defined variable to prefer some python
versions over another...

> - Scripts of packages supporting multiple Python versions cannot be easily (without
> necessity of using of e.g. "python3.1 /usr/bin/${script}") executed with a Python
> version different than active Python version.

If you develop on python then... write your own wrappers or
use /usr/bin/python and that's it! In case you are not I'm not sure why
such switching is useful. Could you elaborate?

--
Peter.
 
Old 12-22-2009, 11:50 PM
Arfrever Frehtes Taifersar Arahesis
 
Default Versioning of Python scripts

2009-12-21 11:50:14 Brian Harring napisał(a):
> On Sat, Dec 19, 2009 at 04:24:49PM +0100, Arfrever Frehtes Taifersar Arahesis wrote:
> > Distutils/Setuptools/Distribute modify shebangs of installed Python scripts, so that they
> > contain path of Python interpreter with version included (e.g. "#!/usr/bin/python3.2").
> > This behavior has both advantage and disadvantages:
> > - Scripts of packages supporting only e.g. Python 2 can be executed (without necessity
> > of using of e.g. "python2 /usr/bin/${script}") after activating of e.g. Python 3.
> > - Scripts of packages supporting multiple Python versions ignore active Python version.
> > - Scripts of packages supporting multiple Python versions cannot be easily (without
> > necessity of using of e.g. "python3.1 /usr/bin/${script}") executed with a Python
> > version different than active Python version.
> > The best solution, which removes these 2 disadvantages and preserves the advantage, seems
> > to be to rename Python scripts to include Python version [1] in filenames, and create wrapper
> > scripts, which call appropriate target scripts [2]. Some files sometimes try to execute
> > e.g. "/usr/bin/python /usr/bin/${script}", so wrapper scripts must be implemented in Python.
> > Wrapper scripts try to execute "${wrapper_script}-${PYTHON_ABI}" files (e.g. "py.test" will
> > execute "py.test-3.1", when Python 3.1 is set as active Python version).
> >
> > distutils.eclass will automatically rename some scripts [3] in "${D}usr/bin" and call
> > the function, which generates wrapper scripts. In case somebody is interested in reading of
> > source code of python_generate_wrapper_scripts() function and potential suggesting of
> > improvements, I'm attaching this function and 2 example wrapper scripts. I'm planning to
> > commit addition of this function in next week.
>
> Not really a huge fan of the EPYTHON var... can you clarify it's real
> world usage?

It simplifies development of ebuilds of packages having non-Distutils-based build systems.
E.g. ebuilds of packages, which support only Python2, install some executables or libraries
(e.g. /usr/lib/kvirc/4.0/modules/libkvipythoncore.so of net-irc/kvirc) linked against
libpythonX.Y.so and don't install any Python modules, will be able to simply call
'python_set_active_version 2', which will export EPYTHON variable with appropriate value.
Ebuilds should never manually set EPYTHON variable.

> I can see that causing all sorts of mayhem as it passes
> it's way down through python scripts invoking other scripts-
> specifically thinking of a py3k only script being forced to 3.1, then
> invoking a py2k script.

EPYTHON variable should be respected only by scripts, which are used during building, testing
or installation of other packages and need to be called with appropriate Python version.
Examples: py.test, trial
distutils_src_install() will call python_generate_wrapper_scripts() (at least by default)
without -E option.

> Beyond that, please provide a way to *disable* this for a pkg.

OK.

--
Arfrever Frehtes Taifersar Arahesis
 

Thread Tools




All times are GMT. The time now is 01:15 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org