On Sun, 2008-05-25 at 08:40 -0700, Russ Allbery wrote:
> Neil Williams <codehelp@debian.org> writes:
> > On Sun, 2008-05-25 at 13:19 +0100, Mark Brown wrote:
>
> >> If you run autotools at build time you should also ensure that the
> >> changes which autotools makes are reverted in the clean target. This
> >> means that your diff doesn't get cluttered with automatically generated
> >> things and ensures that repeated builds of the package produce the same
> >> diff.gz.
>
> > I haven't seen any other packages doing that - is there an example
> > involving aclocal.m4 somewhere?
>
> Lots of other packages do this -- one of mine off the top of my head is
> xml-security-c.

Nope. No mention of aclocal.m4 in debian/rules for that package,
just /usr/share/misc/config.guess and config.sub.

I need to patch configure and configure.ac in this package, that means
that aclocal.m4 is constantly being changed and reverted. It shouldn't
matter - it really should not.

I see no purpose in lintian (or dpkg come to that) testing for changes
in aclocal.m4 - IMHO it should simply be exempt from this check. No
distro can risk patching aclocal.m4 - by it's nature it is transient and
volatile and subject to large changes entirely at the mercy of external
factors. Any build system that tries is simply going to break
constantly, AFAICT.

No matter how I wrap the cp foo foo.safe mv foo.safe foo in
debian/rules, the first build can be OK but all subsequent builds end up
with aclocal.m4 in the .diff.gz. Besides, replacing aclocal.m4 after it
has just been updated by ./configure is not a good idea to me.

> > I really don't want to do all that for the tmpl/* files as well - I
> > don't see the need, copying dozens of files into foo.safe or
> > foo.upstream and then moving them back?
>
> Just delete them then.

??? That simply does not work. The problem is that running gtk-doc not
only requires tmpl/*.sgml files to exist but it *then modifies them*!
Even though I don't install these files into the package, I cannot
delete them and I cannot move them out of the way or the documentation
is not updated. Again, first build can be OK, second build generates a
dozen spurious warnings (because the files are modified after
the .diff.gz is created but cannot be reliably reverted before the next
build).

I don't see that I should provide out-dated documentation (by dropping
--enable-gtk-doc) just because it suits lintian - neither can I patch
the updates because the updates themselves are generated by a third
party tool so I can neither control the changes made nor maintain
"clean" patches. I suspect the issue arises because upstream happen to
prepare the release tarballs on Ubuntu or Fedora where the tool version
differs. The precise reason is inconsequential - the problem is that
Debian should not need to care about these modifications. It's taking
'divergence' one stage too far.

With these gtk-doc files, it's not so much that the tmpl/*.sgml files
are generated but that a tool essential to the build modifies them in a
way that cannot be patched because the results of the patches are
variable according to that third party tool. The changes then affect the
files that are packaged. Some of these are formatting changes -
whitespace changes, extra lines, comment changes. Other changes cause
sections or entire pages to migrate within the final files. The "sense"
of the docs doesn't appear to change, just the internal structure - as
determined by the differing versions of the tools used.

The CDBS build doesn't do anything different - it's just that lintian
doesn't produces any warnings for this check in CDBS packages, ever.

> The point is to not put mechanically generated changes in the diff, since
> it makes it much harder to review what the maintainer has actually done.

I don't see how it can be prevented. Take a look at the current .diff.gz
for libgpewidget 0.115-5 in the archive.

> I think most people take the approach of just deleting such files in the
> clean target, which will exclude their changes from the diff.

Sorry, it is not as simple as that - the package simply won't build if
tmpl/*.sgml are removed, no rule to make tmpl/*.sgml.

The tmpl/*.sgml files cannot be cleaned and removing aclocal.m4 is
simply pointless (and creates an even larger .diff.gz).

I still don't see what problem this test is trying to solve - AFAICT the
problem didn't exist in the first place and all this extra work appears
pointless.

I still think patch-system-but-direct-changes-in-diff should be
downgraded to info only, if kept at all.

I fail to see what I'm doing wrong - or even why it matters to lintian.

We risk swapping a minor problem that only occurs when the maintainer
upgrades from one version to another (and which could be fixed by
dpkg-source ignoring changes to generated files in the .diff.gz or not
applying generated changes that do exist in the .diff.gz) for a major
FTBFS error every time some random third party package is updated by
erroneously patching files that contain generated changes outside the
control of the maintainer.

dpkg already detects whether files have been deleted or modified and
whether those changes can be ignored.

It is impossible to maintain clean patches if those patches must include
generated changes.

A possible solution is to invert this lintian check and say that if a
patch system is in use, dpkg-source refuses to unpack/apply anything
from the .diff.gz that is not in the debian/ directory.

make distclean is fine for compiled object files and .libs stuff but
sometimes it just isn't enough - but equally, I don't see that returning
to some mystical pristine source is valid either.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

On Sun, 2008-05-25 at 08:40 -0700, Russ Allbery wrote:
> Neil Williams <codehelp@debian.org> writes:
> > On Sun, 2008-05-25 at 13:19 +0100, Mark Brown wrote:
>
> >> If you run autotools at build time you should also ensure that the
> >> changes which autotools makes are reverted in the clean target. This
> >> means that your diff doesn't get cluttered with automatically generated
> >> things and ensures that repeated builds of the package produce the same
> >> diff.gz.
>
> > I haven't seen any other packages doing that - is there an example
> > involving aclocal.m4 somewhere?
>
> Lots of other packages do this -- one of mine off the top of my head is
> xml-security-c.

Nope. No mention of aclocal.m4 in debian/rules for that package,
just /usr/share/misc/config.guess and config.sub.

I need to patch configure and configure.ac in this package, that means
that aclocal.m4 is constantly being changed and reverted. It shouldn't
matter - it really should not.

I see no purpose in lintian (or dpkg come to that) testing for changes
in aclocal.m4 - IMHO it should simply be exempt from this check. No
distro can risk patching aclocal.m4 - by it's nature it is transient and
volatile and subject to large changes entirely at the mercy of external
factors. Any build system that tries is simply going to break
constantly, AFAICT.

No matter how I wrap the cp foo foo.safe mv foo.safe foo in
debian/rules, the first build can be OK but all subsequent builds end up
with aclocal.m4 in the .diff.gz. Besides, replacing aclocal.m4 after it
has just been updated by ./configure is not a good idea to me.

> > I really don't want to do all that for the tmpl/* files as well - I
> > don't see the need, copying dozens of files into foo.safe or
> > foo.upstream and then moving them back?
>
> Just delete them then.

??? That simply does not work. The problem is that running gtk-doc not
only requires tmpl/*.sgml files to exist but it *then modifies them*!
Even though I don't install these files into the package, I cannot
delete them and I cannot move them out of the way or the documentation
is not updated. Again, first build can be OK, second build generates a
dozen spurious warnings (because the files are modified after
the .diff.gz is created but cannot be reliably reverted before the next
build).

I don't see that I should provide out-dated documentation (by dropping
--enable-gtk-doc) just because it suits lintian - neither can I patch
the updates because the updates themselves are generated by a third
party tool so I can neither control the changes made nor maintain
"clean" patches. I suspect the issue arises because upstream happen to
prepare the release tarballs on Ubuntu or Fedora where the tool version
differs. The precise reason is inconsequential - the problem is that
Debian should not need to care about these modifications. It's taking
'divergence' one stage too far.

With these gtk-doc files, it's not so much that the tmpl/*.sgml files
are generated but that a tool essential to the build modifies them in a
way that cannot be patched because the results of the patches are
variable according to that third party tool. The changes then affect the
files that are packaged. Some of these are formatting changes -
whitespace changes, extra lines, comment changes. Other changes cause
sections or entire pages to migrate within the final files. The "sense"
of the docs doesn't appear to change, just the internal structure - as
determined by the differing versions of the tools used.

The CDBS build doesn't do anything different - it's just that lintian
doesn't produces any warnings for this check in CDBS packages, ever.

> The point is to not put mechanically generated changes in the diff, since
> it makes it much harder to review what the maintainer has actually done.

I don't see how it can be prevented. Take a look at the current .diff.gz
for libgpewidget 0.115-5 in the archive.

> I think most people take the approach of just deleting such files in the
> clean target, which will exclude their changes from the diff.

Sorry, it is not as simple as that - the package simply won't build if
tmpl/*.sgml are removed, no rule to make tmpl/*.sgml.

The tmpl/*.sgml files cannot be cleaned and removing aclocal.m4 is
simply pointless (and creates an even larger .diff.gz).

I still don't see what problem this test is trying to solve - AFAICT the
problem didn't exist in the first place and all this extra work appears
pointless.

I still think patch-system-but-direct-changes-in-diff should be
downgraded to info only, if kept at all.

I fail to see what I'm doing wrong - or even why it matters to lintian.

We risk swapping a minor problem that only occurs when the maintainer
upgrades from one version to another (and which could be fixed by
dpkg-source ignoring changes to generated files in the .diff.gz or not
applying generated changes that do exist in the .diff.gz) for a major
FTBFS error every time some random third party package is updated by
erroneously patching files that contain generated changes outside the
control of the maintainer.

dpkg already detects whether files have been deleted or modified and
whether those changes can be ignored.

It is impossible to maintain clean patches if those patches must include
generated changes.

A possible solution is to invert this lintian check and say that if a
patch system is in use, dpkg-source refuses to unpack/apply anything
from the .diff.gz that is not in the debian/ directory.

make distclean is fine for compiled object files and .libs stuff but
sometimes it just isn't enough - but equally, I don't see that returning
to some mystical pristine source is valid either.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

Neil Williams <codehelp@debian.org> writes:
> On Sun, 2008-05-25 at 08:40 -0700, Russ Allbery wrote:

>> Lots of other packages do this -- one of mine off the top of my head is
>> xml-security-c.

> Nope. No mention of aclocal.m4 in debian/rules for that package,
> just /usr/share/misc/config.guess and config.sub.

Uh, it's the same problem. Surely the generalization is obvious?

I guess look at shibboleth-sp if it's not....

> I need to patch configure and configure.ac in this package, that means
> that aclocal.m4 is constantly being changed and reverted. It shouldn't
> matter - it really should not.

Right, so delete it in the clean rule.

> I see no purpose in lintian (or dpkg come to that) testing for changes
> in aclocal.m4 - IMHO it should simply be exempt from this check.

Absolutely not -- you're introducing unexpected changes to the packaging
diff file, and that's exactly the point of this check. Delete the
modified and regenerated files in the clean rule and then you won't have
this problem.

> No distro can risk patching aclocal.m4 - by it's nature it is transient
> and volatile and subject to large changes entirely at the mercy of
> external factors. Any build system that tries is simply going to break
> constantly, AFAICT.

Exactly, which is why lintian is making sure that you don't introduce
patches to it in your *.diff.gz inadvertantly, since it's not a sane thing
to do when you're using a patch system. The preferred method to handle it
as far as I'm concerned is doing what you're doing -- running autotools at
build time. In that case, you need to remove the regenerated files in
your clean rule. The other way to do it, particularly if you need a very
specific version of autotools, is to run them yourself and save them as a
patch, but I think that's a bad idea except in very specific situations.

>>> I really don't want to do all that for the tmpl/* files as well - I
>>> don't see the need, copying dozens of files into foo.safe or
>>> foo.upstream and then moving them back?

>> Just delete them then.

> ??? That simply does not work. The problem is that running gtk-doc not
> only requires tmpl/*.sgml files to exist but it *then modifies them*!

This is extremely unusual. Are you *sure* that it does an inplace edit of
the files during the build process? This is almost never what build
systems do; instead, they generate files from other files using templating
systems. They may ship the results of that operation and then redo it
during the build, in which case you need to delete the regenerated files
in your clean rule.

If they really expect the files to exist and to edit them in-place during
the build, upstream is doing something insane, and it's really something
that should be fixed upstream; a lintian warning is drawing your attention
to a very broken behavior.

> Even though I don't install these files into the package, I cannot
> delete them and I cannot move them out of the way or the documentation
> is not updated. Again, first build can be OK, second build generates a
> dozen spurious warnings (because the files are modified after the
> .diff.gz is created but cannot be reliably reverted before the next
> build).

Right, lintian is diagnosing build system brokenness. I'm fairly happy
with lintian's behavior here; what it's drawing attention to is exactly
the kind of thing that breaks repeated package builds or causes NMUs to
introduce spurious package diffs, and is something that should really
ideally be fixed.

> With these gtk-doc files, it's not so much that the tmpl/*.sgml files
> are generated but that a tool essential to the build modifies them in a
> way that cannot be patched because the results of the patches are
> variable according to that third party tool.

If the tool behaves and only behaves in that way (I haven't checked), that
tool is broken and we should fix that tool.

> The CDBS build doesn't do anything different - it's just that lintian
> doesn't produces any warnings for this check in CDBS packages, ever.

Well, yes, it does, if CDBS uses a recognized patch system, but the list
of patch systems that lintian recognizes is fairly limited at this point.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Neil Williams <codehelp@debian.org> writes:
> On Sun, 2008-05-25 at 08:40 -0700, Russ Allbery wrote:

>> Lots of other packages do this -- one of mine off the top of my head is
>> xml-security-c.

> Nope. No mention of aclocal.m4 in debian/rules for that package,
> just /usr/share/misc/config.guess and config.sub.

Uh, it's the same problem. Surely the generalization is obvious?

I guess look at shibboleth-sp if it's not....

> I need to patch configure and configure.ac in this package, that means
> that aclocal.m4 is constantly being changed and reverted. It shouldn't
> matter - it really should not.

Right, so delete it in the clean rule.

> I see no purpose in lintian (or dpkg come to that) testing for changes
> in aclocal.m4 - IMHO it should simply be exempt from this check.

Absolutely not -- you're introducing unexpected changes to the packaging
diff file, and that's exactly the point of this check. Delete the
modified and regenerated files in the clean rule and then you won't have
this problem.

> No distro can risk patching aclocal.m4 - by it's nature it is transient
> and volatile and subject to large changes entirely at the mercy of
> external factors. Any build system that tries is simply going to break
> constantly, AFAICT.

Exactly, which is why lintian is making sure that you don't introduce
patches to it in your *.diff.gz inadvertantly, since it's not a sane thing
to do when you're using a patch system. The preferred method to handle it
as far as I'm concerned is doing what you're doing -- running autotools at
build time. In that case, you need to remove the regenerated files in
your clean rule. The other way to do it, particularly if you need a very
specific version of autotools, is to run them yourself and save them as a
patch, but I think that's a bad idea except in very specific situations.

>>> I really don't want to do all that for the tmpl/* files as well - I
>>> don't see the need, copying dozens of files into foo.safe or
>>> foo.upstream and then moving them back?

>> Just delete them then.

> ??? That simply does not work. The problem is that running gtk-doc not
> only requires tmpl/*.sgml files to exist but it *then modifies them*!

This is extremely unusual. Are you *sure* that it does an inplace edit of
the files during the build process? This is almost never what build
systems do; instead, they generate files from other files using templating
systems. They may ship the results of that operation and then redo it
during the build, in which case you need to delete the regenerated files
in your clean rule.

If they really expect the files to exist and to edit them in-place during
the build, upstream is doing something insane, and it's really something
that should be fixed upstream; a lintian warning is drawing your attention
to a very broken behavior.

> Even though I don't install these files into the package, I cannot
> delete them and I cannot move them out of the way or the documentation
> is not updated. Again, first build can be OK, second build generates a
> dozen spurious warnings (because the files are modified after the
> .diff.gz is created but cannot be reliably reverted before the next
> build).

Right, lintian is diagnosing build system brokenness. I'm fairly happy
with lintian's behavior here; what it's drawing attention to is exactly
the kind of thing that breaks repeated package builds or causes NMUs to
introduce spurious package diffs, and is something that should really
ideally be fixed.

> With these gtk-doc files, it's not so much that the tmpl/*.sgml files
> are generated but that a tool essential to the build modifies them in a
> way that cannot be patched because the results of the patches are
> variable according to that third party tool.

If the tool behaves and only behaves in that way (I haven't checked), that
tool is broken and we should fix that tool.

> The CDBS build doesn't do anything different - it's just that lintian
> doesn't produces any warnings for this check in CDBS packages, ever.

Well, yes, it does, if CDBS uses a recognized patch system, but the list
of patch systems that lintian recognizes is fairly limited at this point.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, May 27, 2008 at 06:06:28PM -0700, Russ Allbery wrote:
> Neil Williams <codehelp@debian.org> writes:

> > With these gtk-doc files, it's not so much that the tmpl/*.sgml files
> > are generated but that a tool essential to the build modifies them in a
> > way that cannot be patched because the results of the patches are
> > variable according to that third party tool.

> If the tool behaves and only behaves in that way (I haven't checked), that
> tool is broken and we should fix that tool.

I've run into a similar situation before with a vaugely borked upstream
build system - it was straightforward enough to work around the problem
by moving the files out of the way and then replacing them if there was
a backup present when clean was run.

--
"You grabbed my hand and we fell into it, like a daydream - or a fever."


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, May 27, 2008 at 06:06:28PM -0700, Russ Allbery wrote:
> Neil Williams <codehelp@debian.org> writes:

> > With these gtk-doc files, it's not so much that the tmpl/*.sgml files
> > are generated but that a tool essential to the build modifies them in a
> > way that cannot be patched because the results of the patches are
> > variable according to that third party tool.

> If the tool behaves and only behaves in that way (I haven't checked), that
> tool is broken and we should fix that tool.

I've run into a similar situation before with a vaugely borked upstream
build system - it was straightforward enough to work around the problem
by moving the files out of the way and then replacing them if there was
a backup present when clean was run.

--
"You grabbed my hand and we fell into it, like a daydream - or a fever."


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, 2008-05-27 at 18:06 -0700, Russ Allbery wrote:
> Neil Williams <codehelp@debian.org> writes:
> > On Sun, 2008-05-25 at 08:40 -0700, Russ Allbery wrote:
>
> >> Lots of other packages do this -- one of mine off the top of my head is
> >> xml-security-c.
>
> > Nope. No mention of aclocal.m4 in debian/rules for that package,
> > just /usr/share/misc/config.guess and config.sub.
>
> Uh, it's the same problem. Surely the generalization is obvious?

It's not quite the same problem because aclocal.m4 is regenerated in the
clean rule itself because changing it causes ./configure --recheck to
recreate the (modified) file. So instead of copying it around, it has to
be deleted - merely because a patch system is in use. Seems crazy to me.

> > I need to patch configure and configure.ac in this package, that means
> > that aclocal.m4 is constantly being changed and reverted. It shouldn't
> > matter - it really should not.
>
> Right, so delete it in the clean rule.

OK. It's a bit like collateral damage - need a patch to configure,
patching causes configure --recheck which modifies aclocal.m4 so delete
aclocal.m4. Hmmm. I don't like it but I'll do it anyway.

> > I see no purpose in lintian (or dpkg come to that) testing for changes
> > in aclocal.m4 - IMHO it should simply be exempt from this check.
>
> Absolutely not -- you're introducing unexpected changes to the packaging
> diff file,

Well my point is that the change is entirely expected (because the
patch modifies a file that is involved in generating the changes) - the
changes are merely a consequence of the patch. It feels wrong to have to
add a clean rule for aclocal.m4 as a direct result of patching
configure.ac when aclocal.m4 was not a problem before the patch.

Anyway, the "problem" of the tmpl/* files is far more resistant.

> >>> I really don't want to do all that for the tmpl/* files as well - I
> >>> don't see the need, copying dozens of files into foo.safe or
> >>> foo.upstream and then moving them back?
>
> >> Just delete them then.
>
> > ??? That simply does not work. The problem is that running gtk-doc not
> > only requires tmpl/*.sgml files to exist but it *then modifies them*!
>
> This is extremely unusual. Are you *sure* that it does an inplace edit of
> the files during the build process?

$ ls libgpewidget-0.115.orig/doc/tmpl/ -1
colordialog.sgml
color-slider.sgml
dirbrowser.sgml
errorbox.sgml
gpeclockface.sgml
gpehelp.sgml
gpeiconlistitem.sgml
gpe-iconlist.sgml
gpeiconlistview.sgml
gpetimesel.sgml
gpewindowlist.sgml
gtkdatecombo.sgml
gtksimplemenu.sgml
infoprint.sgml
init.sgml
libgpewidget-unused.sgml
picturebutton.sgml
pixmaps.sgml
popup_menu.sgml
popup.sgml
question.sgml
smallbox.sgml
spacing.sgml
stylus.sgml
translabel.sgml
tray.sgml
windows.sgml

From the build log:
gtkdoc-mkdb --module=libgpewidget --source-dir=.. --output-format=xml
--sgml-mode --output-format=xml --tmpl-dir=tmpl

Now running lintian...
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/color-slider.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/colordialog.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/colorrenderer.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/gpeclockface.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/gpedialog.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/gpehelp.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/gtkdatecombo.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/gtksimplemenu.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/libgpewidget-unused.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/link-warning.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/pixmaps.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/smallbox.sgml
W: libgpewidget source: patch-system-but-direct-changes-in-diff doc/tmpl/spacing.sgml
Finished running lintian.

13 out of 27 files modified by gtk-doc.

Copy those files back into doc/tmpl/ from the upstream source and they
are still modified at the next build.

Add a rm -f doc/tmpl/*.sgml to clean:: and the build fails: No rule to
make doc/tmp/*.sgml

So, yes, --enable-gtkdoc is modifying files included upstream and which
are essential to the build process.

If I drop --enable-gtkdoc I get different contents in the
libgpewidget-doc package.

> This is almost never what build
> systems do; instead, they generate files from other files using templating
> systems. They may ship the results of that operation and then redo it
> during the build, in which case you need to delete the regenerated files
> in your clean rule.

Yes, that is what we might expect from build systems - unfortunately it
doesn't always work that way.

The way gtk-doc works is that it adds a pre-defined gtk-doc.make file
into the upstream source and it is this file that determines everything
about how gtk-doc works. If upstream are using a different version, I
don't see what can be done. --enable-gtkdoc will update the .make file
according to the installed version on the build system. Disabling gtkdoc
means packaging outdated documentation.

It seems to me you're expecting gtk-doc-tools to implement something
like AM_MAINTAINER_MODE where upstream gtk-doc-tools is allowed to make
changes to doc/tmpl/*.sgml but Debian gtk-doc-tools is not. (I always
thought that AM_MAINTAINER_MODE was considered harmful for precisely
this reason.)

> If they really expect the files to exist and to edit them in-place during
> the build, upstream is doing something insane, and it's really something
> that should be fixed upstream; a lintian warning is drawing your attention
> to a very broken behavior.

AFAICT this is not a fault upstream - the tmpl/*.sgml files need to be
included. I suspect it is merely because upstream don't happen to use
the same version of gtk-doc-tools. The "they" in the quoted paragraph
refers to the gtk-doc-tools upstream, not libgpewidget upstream.

It is not at all insane - every package using gtk-doc-tools has the same
process. I'm not sure why only libgpewidget is currently showing this
problem but it is wrong to call it "insane".

gtk-doc-tools is, in essence, an upstream tool. It is meant to modify
upstream files during the build, that is the reason it exists. The need
to run it in the build arises, I'm presuming, from a difference in the
version of gtk-doc-tools between Debian and the libgpewidget upstream
team. Debian simply cannot stipulate that a certain version must be used
upstream. I don't see the bug.

> Right, lintian is diagnosing build system brokenness. I'm fairly happy
> with lintian's behavior here; what it's drawing attention to is exactly
> the kind of thing that breaks repeated package builds or causes NMUs to
> introduce spurious package diffs, and is something that should really
> ideally be fixed.

Well, I'm not sure that it can be fixed - other than requiring upstream
to use the same version of gtk-doc-tools as Debian which is frankly,
absurd.

> > With these gtk-doc files, it's not so much that the tmpl/*.sgml files
> > are generated but that a tool essential to the build modifies them in a
> > way that cannot be patched because the results of the patches are
> > variable according to that third party tool.
>
> If the tool behaves and only behaves in that way (I haven't checked), that
> tool is broken and we should fix that tool.

It probably is behaving that way but there's no accounting for changes
between versions - or for changes that may yet be made that also tweak
the output. I simply cannot risk patching these tmpl/*.sgml files and
having the patches fail at some point in the future, turning a bug in
lintian into a FTBFS in libgpewidget (which has 40 reverse
dependencies).

I still think that lintian is out of line here - this is not a problem
for the maintainer and it is only the maintainer who needs to worry
about updating the package from 0.115 to 0.116 and 0.117 etc.

These files are in the .diff.gz because of the behaviour of a third
party tool which updates a data source within the upstream source
according to internal rules for that tool. OK, maybe that tool could be
improved but that is not necessarily trivial - especially when Debian
cannot stipulate that any particular upstream team must use a particular
version or configuration for that tool.

I haven't been able to identify the supposed bug in gtk-doc-tools and I
don't see that it is up to me to do that - AFAICT this is something that
is not a problem in the real world and lintian is imposing unrealistic
expectations on libgpewidget.

I'm perfectly happy with the behaviour of gtk-doc-tools in libgpewidget
(I'd like gtk-doc-tools to be workable in a cross-build environment but
that's minor because cross-builds rarely care for API documentation
anyway and it is fairly easy to disable gtk-doc if the docs are not
wanted.). If others disagree, it is up to them to file a bug in
gtk-doc-tools and work out just how it should be improved. The mere fact
of a spurious lintian warning is an annoyance, not a problem but I would
still like lintian to be more intelligent about these things, accept
that the world is not 100% perfect and not blame the messenger.

If this is a bug in libgpewidget that I've missed, please let me know in
the usual manner, but for now I'm just going to ignore lintian for this
check.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

On Wed, 2008-05-28 at 11:45 +0100, Neil Williams wrote:

Ignore the CC for #482716 - the CC should be for #471263
[patch-systems]: please no patch-system-but-direct-changes-in-diff for
generated files


--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/

Neil Williams <codehelp@debian.org> writes:
> On Tue, 2008-05-27 at 18:06 -0700, Russ Allbery wrote:

>> Uh, it's the same problem. Surely the generalization is obvious?

> It's not quite the same problem because aclocal.m4 is regenerated in the
> clean rule itself because changing it causes ./configure --recheck to
> recreate the (modified) file. So instead of copying it around, it has
> to be deleted - merely because a patch system is in use. Seems crazy to
> me.

Yes, deleting it is what I've been trying to tell you to do from the
start. Moving it around is generally a waste of time. The examples
that I've been pointing you to just delete such files.

And no, lintian only happens to *detect* this problem because a patch
system is in use, since otherwise it doesn't know enough. However, you
should *always* have been doing this for exactly the reasons that you
explained in your last message -- patching aclocal.m4 is not sane.
lintian has detected a latent bug in your package now that it has more
information about what's going on.

> OK. It's a bit like collateral damage - need a patch to configure,
> patching causes configure --recheck which modifies aclocal.m4 so delete
> aclocal.m4. Hmmm. I don't like it but I'll do it anyway.

Well, if I were patching configure, I'd run the full autotools suite
myself before running configure rather than relying on configure
--recheck.

>> Absolutely not -- you're introducing unexpected changes to the packaging
>> diff file,

> Well my point is that the change is entirely expected (because the patch
> modifies a file that is involved in generating the changes) - the
> changes are merely a consequence of the patch.

You expect to have a giant patch to generated files in your *.diff.gz of
your source package? I wouldn't -- that's what I meant by unexpected.

> It feels wrong to have to add a clean rule for aclocal.m4 as a direct
> result of patching configure.ac when aclocal.m4 was not a problem before
> the patch.

Well, this is just how autotools work.

> 13 out of 27 files modified by gtk-doc.

What's the diff? What is it doing to the files at build time?

> The way gtk-doc works is that it adds a pre-defined gtk-doc.make file
> into the upstream source and it is this file that determines everything
> about how gtk-doc works. If upstream are using a different version, I
> don't see what can be done. --enable-gtkdoc will update the .make file
> according to the installed version on the build system. Disabling gtkdoc
> means packaging outdated documentation.

I'm not sure how this is related. Is the update of the .make file somehow
causing the changes to the SGML file?

> It seems to me you're expecting gtk-doc-tools to implement something
> like AM_MAINTAINER_MODE where upstream gtk-doc-tools is allowed to make
> changes to doc/tmpl/*.sgml but Debian gtk-doc-tools is not. (I always
> thought that AM_MAINTAINER_MODE was considered harmful for precisely
> this reason.)

No, no, no. I'm expecting something much more simple: any file in a build
tree should be either a generated file or a source file. Surely not both!
A normal build of an upstream package should not modify upstream source
files that have no other origin.

> AFAICT this is not a fault upstream - the tmpl/*.sgml files need to be
> included. I suspect it is merely because upstream don't happen to use
> the same version of gtk-doc-tools. The "they" in the quoted paragraph
> refers to the gtk-doc-tools upstream, not libgpewidget upstream.

Changing the version of a tool should not result in an in-place edit of a
file with no other source. That's just nuts. If those are generated
files that come from some other origin, you should be able to delete them
and regenerate them from scratch. Otherwise, if upstream is shipping
them, it doesn't make sense for a normal build to modify them.

I suspect that there's some way to generate those files that just isn't
part of a normal build, and deleting them really is the right thing to do;
you just need to be more aggressive about regenerating them. But
hopefully someone who knows how gtk-doc-tools work can weigh in here.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

(Please cc me in any replies, I'm not subscribed.)

Neil Williams wrote:
> ??? That simply does not work. The problem is that running gtk-doc not
> only requires tmpl/*.sgml files to exist but it *then modifies them*!

Here's how gtk-doc *used to* work:

* gtk-doc parses source code and writes out skeletal tmpl/*.sgml
* svn ci -m 'initial version of gtkdoc templates' tmpl
* upstream doc author inserts content into tmpl/*.sgml
* svn ci -m 'wrote some docs' tmpl
* gtk-doc parses source code (to pick up any new functions) + tmpl/*.sgml,
and merges them
* svn ci -m 'yay gtkdoc' tmpl
* ...

This is, as you've noticed, insane. Sane upstreams now write all the
content for the documentation in /** */ comments in the source code, so
the tmpl/*.sgml are purely generated and can safely be omitted from
source-code control. (I have no opinion on whether your upstream is sane
or not - please check.)

However, the "no rule to make tmpl/*.sgml" issue still exists, as a
relic of the old build process.

(This history is also why gtkdoc insists on running in $srcdir, even in an
otherwise out-of-tree build - it goes as far as to chmod +w $srcdir to
subvert automake's attempts to catch writes to $srcdir!)

In Telepathy and Farsight we avoid ridiculous conflicts in tmpl/ in our
source-code control as follows:

* put tmpl/*.sgml in .svnignore, .gitignore, the darcs boringfile, or whatever
* as an exception to that, check in an empty file "tmpl/dummy.sgml"
which is not referenced by anything else, so that builds from a clean
checkout will succeed

In theory you could do this to get rid of tmpl/*.sgml in the tarball
too, but gtkdoc.make is in control of "make dist", so you probably can't.

Hope this helps,
Simon


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org