FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-15-2011, 09:29 PM
Marcin Owsiany
 
Default Best practice for cleaning autotools-generated files?

Hello,

The current best practice for dealing with packages using GNU autotools
(as described in /usr/share/doc/autotools-dev/README.Debian.gz) is to
run autoreconf in a prerequisite of a build target, and to remove its
results in the clean target.

However that README does not give any hints on how to best do the
cleaning. How are others doing it?

I could think of the following ways:

* maintain a blacklist of generated files, and "rm" them in the
clean target.

This has the downside that as soon as any of:
- automake
- aclocal
- autoconf
- autopoint
- autoheader
- libtoolize
decides to introduce a new autogenerated file, my package is going to
collect cruft on a sourceful rebuild (e.g. NMUs). And because of the
time skew problems, it is cruft which can potentially break the build
in subtle ways.

Looking (for example) at the amazing list of files that autopoint
copies into the po/ subdirectory, I have very little faith that
something new won't appear in the future.

The only way this would be acceptable was if there was a way to make
dpkg-build abort if added files are found outside the debian
directory. That would cause the person building the package to add
any new files to the blacklist. OTOH the security team probably
wouldn't like it?


* maintain a whitelist of distributed files, and "rm" everything
else (apart from the debian directory) in the clean target.

Since I use (or plan to use) git-buildpackage, I don't have a tarball
which could serve as an authoritative whitelist. Thus an additional
whitelist refresh step would be required every time I merge the
upstream branch into the debian branch. That's bad. Furthermore, a
whitelist approach would mercilessly elliminate all files on a
"clean", so one would have to be really careful not to leave
unchecked but precious files in the source tree at any time... :-/


* for every autoool, maintain an anti-autotool that would know how to
revert the actions of its counterpart. Basically just like
automake-generated files encode the knowledge of how to "make clean"
after a "make all", there would need to be a "-clean" counterpart for
every autotool that autoreconf can call.

However this would need to be a team effort, as such cleanup tools
would need to closely follow their "generator" counterparts.

Are there other ways? Comments?

regards,
--
Marcin Owsiany <porridge@debian.org> http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110315222957.GA5440@beczulka">http://lists.debian.org/20110315222957.GA5440@beczulka
 
Old 03-15-2011, 09:38 PM
Kai Wasserbäch
 
Default Best practice for cleaning autotools-generated files?

Dear Marcin,
Marcin Owsiany schrieb am 15.03.2011 23:29:
> However that README does not give any hints on how to best do the
> cleaning. How are others doing it?

in puf we're using dh-autoreconf in the dh(7) sequence, which creates a list of
modified files, backs them up and restores them later again (through its two
commands dh_autoreconf and dh_autoreconf_clean).

Kind regards,
Kai Wasserbäch



--

Kai Wasserbäch (Kai Wasserbaech)

E-Mail: debian@carbon-project.org
Jabber (debianforum.de): Drizzt
URL: http://wiki.debian.org/C%C3%B9ran
GnuPG: 0xE1DE59D2 0600 96CE F3C8 E733 E5B6 1587 A309 D76C E1DE 59D2
(http://pgpkeys.pca.dfn.de/pks/lookup?search=0xE1DE59D2&fingerprint=on&hash=on&op =vindex)
 
Old 03-15-2011, 09:44 PM
Emilio Pozuelo Monfort
 
Default Best practice for cleaning autotools-generated files?

On 15/03/11 22:29, Marcin Owsiany wrote:
> The current best practice for dealing with packages using GNU autotools
> (as described in /usr/share/doc/autotools-dev/README.Debian.gz) is to
> run autoreconf in a prerequisite of a build target, and to remove its
> results in the clean target.
>
> However that README does not give any hints on how to best do the
> cleaning. How are others doing it?

Just use dh_autoreconf (or autoreconf.mk) from the dh-autoreconf package.

Cheers,
Emilio


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4D7FEBB9.3080609@debian.org">http://lists.debian.org/4D7FEBB9.3080609@debian.org
 
Old 03-15-2011, 09:54 PM
Josue Abarca
 
Default Best practice for cleaning autotools-generated files?

On Tue, Mar 15, 2011 at 10:29:57PM +0000, Marcin Owsiany wrote:
> Hello,
>
> The current best practice for dealing with packages using GNU autotools
> (as described in /usr/share/doc/autotools-dev/README.Debian.gz) is to
> run autoreconf in a prerequisite of a build target, and to remove its
> results in the clean target.
>
> However that README does not give any hints on how to best do the
> cleaning. How are others doing it?
>
...

What about?:

From: /usr/share/doc/autotools-dev/README.Debian.gz
"Example autogen.sh and debian/rules files can be found in
/usr/share/doc/autotools-dev/examples. Do not use them as-is. Rather,
properly customize your own."

and from /usr/share/doc/autotools-dev/examples/autogen.sh
" ...Still, this script is capable of cleaning
# just about any possible mess of autoconf files."

Cheers.
--
Josué M. Abarca S.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110315225446.GD2792@numenor.numenor">http://lists.debian.org/20110315225446.GD2792@numenor.numenor
 
Old 03-15-2011, 09:55 PM
Neil Williams
 
Default Best practice for cleaning autotools-generated files?

On Tue, 15 Mar 2011 22:29:57 +0000
Marcin Owsiany <porridge@debian.org> wrote:

> The current best practice for dealing with packages using GNU
> autotools (as described
> in /usr/share/doc/autotools-dev/README.Debian.gz) is to run
> autoreconf in a prerequisite of a build target, and to remove its
> results in the clean target.
>
> However that README does not give any hints on how to best do the
> cleaning. How are others doing it?

Not every generated file needs to be cleaned. The list can vary if the
package is very old and depends on how the package itself handles the
clean internally.

I'm upstream for a few packages which work this way, so I ensure that
make distclean works correctly before every release, I use CDBS for the
Debian build system working from the make dist tarball, build with
svn-buildpackage, I use svn:ignore properties for some other files and
then the problem just goes away.

> I could think of the following ways:
>
> * maintain a blacklist of generated files, and "rm" them in the
> clean target.

A selective list is fine.

> Looking (for example) at the amazing list of files that autopoint
> copies into the po/ subdirectory, I have very little faith that
> something new won't appear in the future.

Not all those files matter. Besides, I've just done autoreconf on one
of my upstream packages with lots of translations and autopoint didn't
add any files - it's all just .po, .gmo, POTFILES* and Make*.

> Since I use (or plan to use) git-buildpackage, I don't have a
> tarball which could serve as an authoritative whitelist. Thus an
> additional whitelist refresh step would be required every time I
> merge the upstream branch into the debian branch. That's bad.

Is this a case of a problem with a tool causing more work? Fix the
tool? Change your choice of tool?

Other tools, like svn-buildpackage, don't have this problem, via the
mergeWithUpstream support and/or a ../tarballs/ directory. Not perfect
but it works.

cvs-buildpackage has a different problem (only with native packages
using autotools) which bites me once in a while but not often enough to
make me actually migrate the package concerned to svn. So I guess it's
a matter of how much of an itch you can withstand.

Using autotools with Debian native packages can be a real PITA and
source format 3.0 just gets it wrong mostly. Most other tools also get
it wrong, even svn-buildpackage. If I knew how to fix it reliably -
*without* putting huge amounts of stuff into the VCS or massively
extending the build by not using make dist (and missing errors by not
using make distcheck), I'd stop using my own hand-crafted scripts to
release these projects. Makes me wish I could pretend that these really
are upstream packages.

I think CMake makes a better job of native packages which need
autotools-type hand-holding, so I may port to that instead.

In other words - fix the tool(s), work with upstream and don't try to
make life difficult for yourself (or people like me who really don't
need this kind of thing to become Policy).

Building direct from VCS is a nice idea but sometimes, it just isn't
worth the pain - let the build system build the tarball and package
from the tarball. It's just easier. Or just change the build system
and/or the tool.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/
 
Old 03-15-2011, 10:35 PM
Ben Hutchings
 
Default Best practice for cleaning autotools-generated files?

On Tue, Mar 15, 2011 at 10:55:54PM +0000, Neil Williams wrote:
> On Tue, 15 Mar 2011 22:29:57 +0000
> Marcin Owsiany <porridge@debian.org> wrote:
[...]
> > Since I use (or plan to use) git-buildpackage, I don't have a
> > tarball which could serve as an authoritative whitelist. Thus an
> > additional whitelist refresh step would be required every time I
> > merge the upstream branch into the debian branch. That's bad.
>
> Is this a case of a problem with a tool causing more work? Fix the
> tool? Change your choice of tool?

Yes, autotools indeed seems to require far more work than it saves.
(I know that's not what you meant, but it is the cause of this
problem.)

> Other tools, like svn-buildpackage, don't have this problem, via the
> mergeWithUpstream support and/or a ../tarballs/ directory. Not perfect
> but it works.
[...]

mergeUpstream works only if you never want to look at upstream changes.

Ben.

--
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert Camus


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110315233559.GC2356@decadent.org.uk">http://lists.debian.org/20110315233559.GC2356@decadent.org.uk
 
Old 03-16-2011, 07:15 AM
Niels Thykier
 
Default Best practice for cleaning autotools-generated files?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-03-15 23:55, Neil Williams wrote:
> Using autotools with Debian native packages can be a real PITA and
> source format 3.0 just gets it wrong mostly. Most other tools also get
> it wrong, even svn-buildpackage. If I knew how to fix it reliably -
> *without* putting huge amounts of stuff into the VCS or massively
> extending the build by not using make dist (and missing errors by not
> using make distcheck), I'd stop using my own hand-crafted scripts to
> release these projects. Makes me wish I could pretend that these really
> are upstream packages.
>

While I have not used it myself, it seems dpkg-source can (partly) help
you here by using the "extend-diff-ignore" option in
debian/source/options.[1]

~Niels

[1]
http://raphaelhertzog.com/2011/01/28/3-ways-to-not-clutter-your-debian-source-package-with-autogenerated-files/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNgHG3AAoJEAVLu599gGRCAuMQAKthJyLrpW 9PUQNc11G5x/tw
emhRttTwaT7Xz9gmpQg6Wp5rFmFCSDjJjZ8bAa/e5G8YyeoXXjsGsK6fN0bjdu8O
8TaRBFlMpimnmfKH2ONfwqSS2chib8CK55vKGJ6ol0TDNqj9TT/xJCPVTKWbkbem
IYACZPha5i0yqlbd0DdN/OrpJtyHlIhVTmHDFdS93sjc/yA2oxJ4dSiZ8J/Exqki
arKJe7K3g1xrInb8U/Z8i8eL7zGiPsq8KFsZEkByGipXGEYsCMz6yi8JS/048S8c
pNy9Jkw6o/bC98FhFfLsgdy8T1gtqKKoVs5ad+9Vl7YI5ssjzJRP0BEsR/EKDfxP
11q8EjIIsxw4L/+IXmUPsGO5UBnAwWkKWJYuVCqga2IjM6NK5Hpqr4u9laJi805f
RjPIjo541kDm3TmShXLFfJJNVSBp+lPr+3u0Yo229P08UpgDf8 4wfXE3I27IJS3j
M0wivqz5rGkfMLK+lyv9Fso5GQmVydrqhxVAHo/9nQCZi0Xz5u3iGjXhNBrORr9M
CfK18MQsWe83XPtftsWcMtGiU4gMg0kfAzkzT7Y4UAtQKU6FKI QSX7i4oT/DSJuN
IaIoo6HB7d6r0Ay+QlASiuneXtU2zHwL2iWFr7GZhmTQbfoE7l MXAnDWNjiJ9CvN
kQrkHxAwFt65wvRZ60E0
=Rd7x
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4D8071B8.6000309@thykier.net">http://lists.debian.org/4D8071B8.6000309@thykier.net
 
Old 03-16-2011, 07:54 AM
sean finney
 
Default Best practice for cleaning autotools-generated files?

On Tue, Mar 15, 2011 at 10:29:57PM +0000, Marcin Owsiany wrote:
> * maintain a whitelist of distributed files, and "rm" everything
> else (apart from the debian directory) in the clean target.
>
> Since I use (or plan to use) git-buildpackage, I don't have a tarball
> which could serve as an authoritative whitelist. Thus an additional
> whitelist refresh step would be required every time I merge the
> upstream branch into the debian branch. That's bad. Furthermore, a
> whitelist approach would mercilessly elliminate all files on a
> "clean", so one would have to be really careful not to leave
> unchecked but precious files in the source tree at any time... :-/

while i've heard that the latest dh/cdbs can do this automatically for
you, in a few packages that I maintain where they're still running
dh 5, i keep an AUTOFOO_CRUFT variable in debian rules, and have a small
amount of code in the corresponding build and clean areas to "preserve"
and "unpreserve" these files. Example:

http://git.debian.org/?p=pkg-xorg/app/compiz.git;a=blob;f=debian/rules;h=530ff40f4e5228a6207d9d6d5a0f8a68acecc0b9;h b=a481aecfeea68adea9d0579059c8eb3a746fcd44

it's not perfect, but seems to do the trick. note that this is not a
list of "everythign autofoo creates/modifies", but rather a list
of "everythign that make distclean fails to remove/restore". It tends
to gather files over time and some of them might no longer apply
in later upstream releases, though there's no harm done apart from
maybe a couple wasted cp operations.

On Tue, Mar 15, 2011 at 10:55:54PM +0000, Neil Williams wrote:
> Other tools, like svn-buildpackage, don't have this problem, via the
> mergeWithUpstream support and/or a ../tarballs/ directory. Not perfect
> but it works.

no, they do have this problem but most people just choose to ignore it
until it causes an FTBFS. If you build with mergeWithUpstream there is
the same likelihood that cruft gets left around, it just doesn't dirty
your working directory/checkout. You can still run into problems if
someone downloads the source package from a mirror and tries to a
build/clean/build.

> I think CMake makes a better job of native packages which need
> autotools-type hand-holding, so I may port to that instead.

IIRC CMake for the most part generates its cruft in a nukable
subdirectory, which makes cleanup significantly simpler.

> Building direct from VCS is a nice idea but sometimes, it just isn't
> worth the pain - let the build system build the tarball and package
> from the tarball. It's just easier. Or just change the build system
> and/or the tool.

IYHO anyway -- I and I'm sure others would probably beg to differ


sean


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110316085426.GA12644@cobija.connexer.com">http://lists.debian.org/20110316085426.GA12644@cobija.connexer.com
 
Old 03-16-2011, 08:20 AM
Neil Williams
 
Default Best practice for cleaning autotools-generated files?

On Tue, 15 Mar 2011 23:35:59 +0000
Ben Hutchings <ben@decadent.org.uk> wrote:

> On Tue, Mar 15, 2011 at 10:55:54PM +0000, Neil Williams wrote:
> > Other tools, like svn-buildpackage, don't have this problem, via the
> > mergeWithUpstream support and/or a ../tarballs/ directory. Not perfect
> > but it works.
> [...]
>
> mergeUpstream works only if you never want to look at upstream changes.

Personally, I'll use debdiff on the .dsc to view changes of all types,
prior to further testing / installation / pbuilder / upload etc. The
upstream changes are available and, in some ways, having a
separate ../tarballs/ directory means that I can unpack
both .orig.tar.bz2's and compare the upstream code directly without
needing to think about the effect of Debian changes, if I want to look
at these in isolation.

Generally though, my problems are confined to a single corner case:
native packages using autotools.

It's such a minor irritation for this specific package that I haven't
got the need to change the package build system (the package code is
stable), I haven't even got sufficient need to change the VCS. One day
I'll get around to a proper solution but I am curious about whether
others have similar problems with such packages. It probably comes down
to a bad choice of build system, if I'm honest.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/
 
Old 03-16-2011, 08:30 AM
"Bernhard R. Link"
 
Default Best practice for cleaning autotools-generated files?

* Marcin Owsiany <porridge@debian.org> [110315 23:30]:
> The current best practice for dealing with packages using GNU autotools
> (as described in /usr/share/doc/autotools-dev/README.Debian.gz) is to
> run autoreconf in a prerequisite of a build target, and to remove its
> results in the clean target.

Well, here I need to contradict. As long as upstream's build system
works and does not need any modifications, redoing that with other
versions is just an unneeded modification of upstream sources.

> * for every autoool, maintain an anti-autotool that would know how to
> revert the actions of its counterpart.

I doubt that is feasible. There are things like files where a default
version is installed if there is non of the maintainer. Things like
that cannot be reverted without knowing what the original state was.
And I'm not sure there is nowhere a way to let maintainer-given actions
run at that time.

> Basically just like
> automake-generated files encode the knowledge of how to "make clean"
> after a "make all", there would need to be a "-clean" counterpart for
> every autotool that autoreconf can call.

There is maintainer-clean, but that has semantics that the maintainer
has to decide. This can be "revert to the manually edited files that
are stored in VCS", but also something different.
(For example automake manual also suggests another rule-set where
maintainer-clean should not remove configure).

Bernhard R. Link


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110316093020.GA19855@pcpool00.mathematik.uni-freiburg.de">http://lists.debian.org/20110316093020.GA19855@pcpool00.mathematik.uni-freiburg.de
 

Thread Tools




All times are GMT. The time now is 04:28 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org