FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 02-03-2012, 06:33 PM
Michael Biebl
 
Default Doesn't contain source for waf binary code

Hi,

as this issue affects quite a few packages, I'd like to bring this up
for wider discussion.

The issue basically is, that the waf build system uses a python script,
which embeds a bz2 tarball containing further python sources. Those are
unpacked to .waf-*/ when the waf script is executed. More details can be
found at [1].

A few observations/questions:

* We do accept tarball-in-tarball packages. What makes this specific
case different?

* Where exactly is the DFSG violation? The package does ship the source
code even if embedding it in the waf script doesn't make that obvious

but

* Would it be sufficient to document in README.source how a user can
unpack and modify the waf sources (basically what [2] does) and wouldn't
this fulfill the DFSG requirements?


* What about packages which ship both an autotools and waf based build
system and the Debian package uses the one based on autotools?



Michael


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645190
[2] http://wiki.debian.org/UnpackWaf


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
 
Old 02-07-2012, 12:16 PM
Ian Jackson
 
Default Doesn't contain source for waf binary code

Michael Biebl writes ("Doesn't contain source for waf binary code"):
> as this issue affects quite a few packages, I'd like to bring this up
> for wider discussion.
>
> The issue basically is, that the waf build system uses a python script,
> which embeds a bz2 tarball containing further python sources. Those are
> unpacked to .waf-*/ when the waf script is executed. More details can be
> found at [1].

This is quite astonishing enough, but the situation is in fact even
worse than it appears. I investigated, and my conclusions are:

"waf" is a build system written in Python. It is normally distributed
in the form of a script called "waf", which the waf authors intend for
upstream authors to include in their upstream distribution tarballs
etc. The script is a self-extracting archive which, whenever it's
run, extracts the tarball out of itself into a temporary directory and
then passes control to the python code it has just extracted.

This IMO would be bad enough to reject an upstream package, in this
form, in Debian. After all we want to be able to modify the build
system as well as the package and this approach makes it unreasonably
hard to do so. And if the build system has some kind of bug we don't
want to have to update dozens of copies embedded in individual source
packages.

But there is more, and worse.

I compared the tarball in waf in postler 0.1.1-1.1 with the upstream
code as obtained from "git clone https://code.google.com/p/waf/". It
turns out that the tarball embedded in the "waf" script is not the
original "waf" source distribution. It contains a subset of the
files, and those files it does contain have been processed to remove
comments, whitespace, etc., much like a JavaScript minimisation. Ie
the "waf" self-extracting archive is generated out of the waf.git
source code by massaging some of the files; modified versions of the
script are supposed to be generated by editing the waf.git
distribution and rerunning its build.

This means that we are distributing files derived from the waf.git
source code, but not the waf.git source code itself. This is of
course completely unacceptable in Debian. (It is not a violation of
the copyright on waf itself as waf has a permissive non-copyleft
licence; but will be a breach of the copyright on any GPL'd waf-using
package, because the GPL's requirements extend to the build system.)

I suggest the following fix:

* Upstream waf should be packaged somehow. Upstream's declared
policy of asking packages to ship a copy of waf suggests that there
won't be much API stability so we will need to encode the waf
version number in the package name, and we may need to package
multiple versions of waf.

* All packages which currently use an included copy of waf should be
changed to use a system-provided copy of waf instead.

* We should treat the file "waf" in the root of affected packages as
we do any other file which is non-DFSG-compliant but which we do
have permission to redistribute. Our current practice is to repack
upstream source archives to remove these files from the Debian
sources. (I think this is pointless makework but changing that
policy is out of scope for this discussion.)

* It is possible that some upstream "source" packages contain "waf"
scripts which were generated from modified versions of waf.git. In
this case we may discover that those packages cannot be built with
publicly available versions of waf.git.

For those packages, we need to obtain the relevant version of
waf.git (for example, by hassling the package upstream), or perhaps
if it's a simple change we can create our own suitable version of
waf.git by "decompiling" (de-minimising) the supplied "waf" script
tarball contents (and permanently maintain the forked waf.git).

If neither of those is possible and the issue can't be worked
around some other way we will have to move the affected package
from main to non-free; if the affected upstream package is
copylefted then we will have no permission to distribute it at all
and must drop it entirely.

I think this is a release-critical bug for all the affected packages.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20273.9233.121924.949120@chiark.greenend.org.uk">h ttp://lists.debian.org/20273.9233.121924.949120@chiark.greenend.org.uk
 
Old 02-07-2012, 02:49 PM
Michael Biebl
 
Default Doesn't contain source for waf binary code

On 07.02.2012 14:16, Ian Jackson wrote:

> I compared the tarball in waf in postler 0.1.1-1.1 with the upstream
> code as obtained from "git clone https://code.google.com/p/waf/". It
> turns out that the tarball embedded in the "waf" script is not the
> original "waf" source distribution. It contains a subset of the
> files, and those files it does contain have been processed to remove
> comments, whitespace, etc., much like a JavaScript minimisation. Ie
> the "waf" self-extracting archive is generated out of the waf.git
> source code by massaging some of the files; modified versions of the
> script are supposed to be generated by editing the waf.git
> distribution and rerunning its build.

The waf-unpack tagged bug reports refer to the Debian wiki with
instructions how to ship the waf script in unpacked form [2].

I assume the FTP team at least considers this approach as acceptable and
quite of few of the affected packages have been "fixed" this way already.

Michael

[1]
http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=ftpmaster@debian.org;tag=waf-unpack
[2] http://wiki.debian.org/UnpackWaf
[3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654500#25
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
 
Old 02-07-2012, 03:19 PM
Ian Jackson
 
Default Doesn't contain source for waf binary code

Michael Biebl writes ("Re: Doesn't contain source for waf binary code"):
> The waf-unpack tagged bug reports refer to the Debian wiki with
> instructions how to ship the waf script in unpacked form [2].

This is, unfortunately, not sufficient.

> I assume the FTP team at least considers this approach as acceptable and
> quite of few of the affected packages have been "fixed" this way already.

The FTP team probably assumed that the rather hard to read Python code
in the "unpacked" waf was actually the source code. However, it is
not.

It is disappointing that none of the people involved with packaging
waf-using software spotted this problem.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20273.20260.603921.659142@chiark.greenend.org.uk"> http://lists.debian.org/20273.20260.603921.659142@chiark.greenend.org.uk
 
Old 02-07-2012, 03:25 PM
Alexander Reichle-Schmehl
 
Default Doesn't contain source for waf binary code

Hi!


Am 07.02.2012 14:16, schrieb Ian Jackson:

> * Upstream waf should be packaged somehow. Upstream's declared
> policy of asking packages to ship a copy of waf suggests that there
> won't be much API stability so we will need to encode the waf
> version number in the package name, and we may need to package
> multiple versions of waf.

Sorry, I'm quite busy and don't have time for a longer answer. We'll
follow up on this soonish.

However, regarding that specific point: waf once was packaged in
Debian. See <20100227195857.07540195@utumno>
(http://lists.debian.org/debian-devel/2010/02/msg00714.html) for details
about the removal.


Best regards,
Alexander


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F31508C.2060805@debian.org">http://lists.debian.org/4F31508C.2060805@debian.org
 
Old 02-07-2012, 03:33 PM
Jerome BENOIT
 
Default Doesn't contain source for waf binary code

Hello List:

what is the status of an orphaned package but that is waiting for an sponsor to be uploaded ?

Thanks,
Jerome

On 03/02/12 20:33, Michael Biebl wrote:

Hi,

as this issue affects quite a few packages, I'd like to bring this up
for wider discussion.

The issue basically is, that the waf build system uses a python script,
which embeds a bz2 tarball containing further python sources. Those are
unpacked to .waf-*/ when the waf script is executed. More details can be
found at [1].

A few observations/questions:

* We do accept tarball-in-tarball packages. What makes this specific
case different?

* Where exactly is the DFSG violation? The package does ship the source
code even if embedding it in the waf script doesn't make that obvious

but

* Would it be sufficient to document in README.source how a user can
unpack and modify the waf sources (basically what [2] does) and wouldn't
this fulfill the DFSG requirements?


* What about packages which ship both an autotools and waf based build
system and the Debian package uses the one based on autotools?



Michael


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645190
[2] http://wiki.debian.org/UnpackWaf





--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F315275.4070805@rezozer.net">http://lists.debian.org/4F315275.4070805@rezozer.net
 
Old 02-07-2012, 03:44 PM
Roger Leigh
 
Default Doesn't contain source for waf binary code

On Tue, Feb 07, 2012 at 01:16:01PM +0000, Ian Jackson wrote:
> Michael Biebl writes ("Doesn't contain source for waf binary code"):
> > as this issue affects quite a few packages, I'd like to bring this up
> > for wider discussion.
> >
> > The issue basically is, that the waf build system uses a python script,
> > which embeds a bz2 tarball containing further python sources. Those are
> > unpacked to .waf-*/ when the waf script is executed. More details can be
> > found at [1].
>
> This means that we are distributing files derived from the waf.git
> source code, but not the waf.git source code itself. This is of
> course completely unacceptable in Debian. (It is not a violation of
> the copyright on waf itself as waf has a permissive non-copyleft
> licence; but will be a breach of the copyright on any GPL'd waf-using
> package, because the GPL's requirements extend to the build system.)

While the waf behavour does sound quite awful, is this really
any different than the current behaviour of the autotools? The
"configure" script is an unreadable mess generated by the
expansion of macros in the autotools packages; it too bears little
relation to the original macros.

(This one of the reasons I hope autoconf will use shell functions in
the future.)


Regards,
Roger

--
.'`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120207164410.GF8832@codelibre.net">http://lists.debian.org/20120207164410.GF8832@codelibre.net
 
Old 02-07-2012, 03:48 PM
Jerome BENOIT
 
Default Doesn't contain source for waf binary code

Sorry, this message was meant for an other threat.

On 07/02/12 17:33, Jerome BENOIT wrote:

Hello List:

what is the status of an orphaned package but that is waiting for an sponsor to be uploaded ?

Thanks,
Jerome

On 03/02/12 20:33, Michael Biebl wrote:

Hi,

as this issue affects quite a few packages, I'd like to bring this up
for wider discussion.

The issue basically is, that the waf build system uses a python script,
which embeds a bz2 tarball containing further python sources. Those are
unpacked to .waf-*/ when the waf script is executed. More details can be
found at [1].

A few observations/questions:

* We do accept tarball-in-tarball packages. What makes this specific
case different?

* Where exactly is the DFSG violation? The package does ship the source
code even if embedding it in the waf script doesn't make that obvious

but

* Would it be sufficient to document in README.source how a user can
unpack and modify the waf sources (basically what [2] does) and wouldn't
this fulfill the DFSG requirements?


* What about packages which ship both an autotools and waf based build
system and the Debian package uses the one based on autotools?



Michael


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645190
[2] http://wiki.debian.org/UnpackWaf








--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F3155EB.1060103@rezozer.net">http://lists.debian.org/4F3155EB.1060103@rezozer.net
 
Old 02-07-2012, 03:51 PM
Jelmer Vernooij
 
Default Doesn't contain source for waf binary code

On 02/07/2012 05:19 PM, Ian Jackson wrote:

Michael Biebl writes ("Re: Doesn't contain source for waf binary code"):

The waf-unpack tagged bug reports refer to the Debian wiki with
instructions how to ship the waf script in unpacked form [2].

This is, unfortunately, not sufficient.


I assume the FTP team at least considers this approach as acceptable and
quite of few of the affected packages have been "fixed" this way already.

The FTP team probably assumed that the rather hard to read Python code
in the "unpacked" waf was actually the source code. However, it is
not.

Samba ships a snapshot of the upstream "wafadmin" python package and the
waf-light script. They can be found as 'wafadmin' and 'waf-light' in
https://code.google.com/p/waf.waf15/.


As far as I can tell there is no functional difference between using
waf-light and the 'waf' script (direct or unpacked).


This is the code that the upstream developers directly work on, and it
is what is used as the basis for generating the
stripped-code-in-a-tarball-in-python minimal "waf" script.


Cheers,

Jelmer


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F31569B.40905@samba.org">http://lists.debian.org/4F31569B.40905@samba.org
 
Old 02-07-2012, 04:12 PM
Russ Allbery
 
Default Doesn't contain source for waf binary code

Roger Leigh <rleigh@codelibre.net> writes:

> While the waf behavour does sound quite awful, is this really any
> different than the current behaviour of the autotools? The "configure"
> script is an unreadable mess generated by the expansion of macros in the
> autotools packages; it too bears little relation to the original macros.

But all the original macros are actually present and shipped with the
source (or are obtained from some version of the Autoconf, Automake, and
Libtool packages), and I think that failure to regenerate the configure
machinery with current Autotools would constitute a bug. The difference
with waf is that the sources aren't included and aren't easily obtainable
via another package in Debian one can build-depend on.

I'm converting all of my packages to use dh-autoreconf so that I can
detect such bugs.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87r4y636uy.fsf@windlord.stanford.edu">http://lists.debian.org/87r4y636uy.fsf@windlord.stanford.edu
 

Thread Tools




All times are GMT. The time now is 12:13 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org