FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 05-29-2011, 08:53 AM
Raphael Hertzog
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Hello,

from time to time I hear some rumblings about how "3.0 (quilt)" mixes
badly with VCS. Indeed, one of the primary goals of the format
was to not require prior knowledge of the patch system to be able to
modify a package. And it's the case since you can do:
- dpkg-source -x
- modify files
- debuild

But there are a few downsides:

Auto-application of patches
---------------------------

Since dpkg-source -x auto-applies the patch, the build part was
designed with the assumption that patches are pre-applied and this
assumption is often wrong for people using VCS. I have resolved this
by auto-applying the patch at the start of a build but (some) users of VCS
have been annoyed with it because it left the VCS unclean after the
build. The option --unapply-patches has been added as a way to change
the behaviour (and you can put it debian/source/local-options).

I would like to improve this situation and not force the majority of
people to add the unapply-patches option (if it turns out the majority
of people use this option or are annoyed by the patches being applied).

I see 2 ways to solve this:
a/ detect the common VCS and make --unapply-patches the default in that
case (but it would require a --no-unapply-patches for the people who
keep the patches applied in their VCS)
b/ modify dpkg-source --before-build to keep a trace of the fact that
it applied the patches (for example by creating
.pc/dpkg-source-auto-applied) and in that case have dpkg-source
--after-build unapply the patches so that we're back to a clean
state after a succesful build.
If the build fails, we'd keep the patches applied.

My preference goes to b/ because it doesn't require changes for people
who like to keep the patches applied in their VCS too. And it's the
principle of least surprise, you keep the same state afer a build than
you had before the build (so it's still ok for people who rely
on the scenario unpack/hack/rebuild).

Comments? Does this look reasonable?


Automatic patches
-----------------

Again to cope with the scenario explained at the start of this mail,
once a user has made modifications we must ensure that they end up in a
proper patch in debian/patches/. Right now this is entirely automatic,
the generated patches take the form of
debian/patches/debian-changes-<ver>.

Obviously this is a pretty poor name for a patch and given it's
automatically generated, it doesn't contain much useful description.
In general they are renamed by the maintainer who also adds a
description (or they are properly put in place before the build so that
dpkg-source doesn't have to create it).

But it still happens that those patches are generated[1] when the maintainer
did not expect any change at all. That's why we added the option
--abort-on-upstream-changes for maintainers who never wants dpkg-source
to auto-create a patch.

I wonder if I should not make this option the default and fail with an
error messages suggesting that the maintainer should run "dpkg-source
--record-changes" if he really wants to keep the current changes. It
would also leave the diff around somewhere in $TMPDIR if the user wants
to review it.

It would break the initial scenario but since the solution is given in the
error message, I believe it would be acceptable.

"dpkg-source --record-changes <patch-name>" would be an interactive command
because you would be asked to fill the patch description header.

We still have the case of people who keep all their changes in the VCS
without maintaining debian/patches/ and who uses --single-debian-patch.
In that case, --abort-on-upstream-changes would not apply.

What do you think of this? Do you have suggestions to improve this
workflow?

Ideally, if desired, dpkg-source --record-changes could be automatically
called in the build process but I'm wary of introducing something
interactive in the package build process.

Cheers,

[1] http://lintian.debian.org/tags/format-3.0-but-debian-changes-patch.html
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110529085303.GA17707@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110529085303.GA17707@rivendell.home.ouaza.com
 
Old 05-29-2011, 09:26 AM
Josselin Mouette
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Le dimanche 29 mai 2011 * 10:53 +0200, Raphael Hertzog a écrit :
> b/ modify dpkg-source --before-build to keep a trace of the fact that
> it applied the patches (for example by creating
> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
> --after-build unapply the patches so that we're back to a clean
> state after a succesful build.
> If the build fails, we'd keep the patches applied.
>
> My preference goes to b/ because it doesn't require changes for people
> who like to keep the patches applied in their VCS too. And it's the
> principle of least surprise, you keep the same state afer a build than
> you had before the build (so it's still ok for people who rely
> on the scenario unpack/hack/rebuild).

I’m not fond of the idea of having dpkg behave differently based on the
presence of a VCS. This should be orthogonal, especially given that
people have different usage patterns for their VCS.

Hence b/ looks much more reasonable.

> But it still happens that those patches are generated[1] when the maintainer
> did not expect any change at all. That's why we added the option
> --abort-on-upstream-changes for maintainers who never wants dpkg-source
> to auto-create a patch.
>
> I wonder if I should not make this option the default

Yes please.

--
.'`. Josselin Mouette
: :' :
`. `'
`-
 
Old 05-29-2011, 09:28 AM
Cyril Brulebois
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Hi,

Raphael Hertzog <hertzog@debian.org> (29/05/2011):
> from time to time I hear some rumblings about how "3.0 (quilt)"
> mixes badly with VCS. Indeed, one of the primary goals of the format
> was to not require prior knowledge of the patch system to be able to
> modify a package.

thanks for trying to improve the situation.

> b/ modify dpkg-source --before-build to keep a trace of the fact that
> it applied the patches (for example by creating
> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
> --after-build unapply the patches so that we're back to a clean
> state after a succesful build.
> If the build fails, we'd keep the patches applied.
>
> My preference goes to b/ because it doesn't require changes for
> people who like to keep the patches applied in their VCS too. And
> it's the principle of least surprise, you keep the same state afer a
> build than you had before the build (so it's still ok for people who
> rely on the scenario unpack/hack/rebuild).
>
> Comments? Does this look reasonable?

I think that from 1.0 format, a use case is still broken with that
approach:
- build a package
- install its binaries
- check the behaviour is the same as with the package in the archive
(i.e. rebuilding doesn't fix the issue magically)
- hack hack hack
- debuild|dpkg-buildpackage -b -nc
→ oh, files were patched/unpatched/changed, you get to rebuild a lot
of things, instead of just the relevant files

Unfortunately I come empty-handed, but I wanted to mention that use
case anyway.

Mraw,
KiBi.
 
Old 05-29-2011, 12:09 PM
Benjamin Drung
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Am Sonntag, den 29.05.2011, 10:53 +0200 schrieb Raphael Hertzog:
> Again to cope with the scenario explained at the start of this mail,
> once a user has made modifications we must ensure that they end up in a
> proper patch in debian/patches/. Right now this is entirely automatic,
> the generated patches take the form of
> debian/patches/debian-changes-<ver>.
>
> Obviously this is a pretty poor name for a patch [...]

The file should end with .patch
(debian/patches/debian-changes-<ver>.patch) so that your favorite text
editor uses the correct highlighting.

--
Benjamin Drung
Debian & Ubuntu Developer
 
Old 05-29-2011, 12:45 PM
"Bernhard R. Link"
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

* Raphael Hertzog <hertzog@debian.org> [110529 10:53]:
> I see 2 ways to solve this:
> a/ detect the common VCS and make --unapply-patches the default in that
> case (but it would require a --no-unapply-patches for the people who
> keep the patches applied in their VCS)

I'd be very disappointed if the more consistent way of having patches
applied in vcs would be punished this way just because some 'majority'
uses their vcs like svn was still state of the art.

> b/ modify dpkg-source --before-build to keep a trace of the fact that
> it applied the patches (for example by creating
> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
> --after-build unapply the patches so that we're back to a clean
> state after a succesful build.
> If the build fails, we'd keep the patches applied.

That sounds a bit better, but it adds even more magic to dpkg-source.
I really miss some way to express: "In this account, do not use magic.
If things are not correct and need fixing, tell me what is wrong and
abort so I'll never miss it." (Actually I'd prefer it as default and
only have it enabled by some options, but a way to globally disable
and turn them into hard errors would would be good enough[tm].)

> But it still happens that those patches are generated[1] when the maintainer
> did not expect any change at all. That's why we added the option
> --abort-on-upstream-changes for maintainers who never wants dpkg-source
> to auto-create a patch.

There is still no way to get this behaviour account-wide, is there?

> I wonder if I should not make this option the default and fail with an
> error messages suggesting that the maintainer should run "dpkg-source
> --record-changes" if he really wants to keep the current changes. It
> would also leave the diff around somewhere in $TMPDIR if the user wants
> to review it.

Sounds nice.

Bernhard R. Link


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110529124544.GA9602@pcpool00.mathematik.uni-freiburg.de">http://lists.debian.org/20110529124544.GA9602@pcpool00.mathematik.uni-freiburg.de
 
Old 05-29-2011, 01:17 PM
Goswin von Brederlow
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Cyril Brulebois <kibi@debian.org> writes:

> Hi,
>
> Raphael Hertzog <hertzog@debian.org> (29/05/2011):
>> from time to time I hear some rumblings about how "3.0 (quilt)"
>> mixes badly with VCS. Indeed, one of the primary goals of the format
>> was to not require prior knowledge of the patch system to be able to
>> modify a package.
>
> thanks for trying to improve the situation.
>
>> b/ modify dpkg-source --before-build to keep a trace of the fact that
>> it applied the patches (for example by creating
>> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
>> --after-build unapply the patches so that we're back to a clean
>> state after a succesful build.
>> If the build fails, we'd keep the patches applied.
>>
>> My preference goes to b/ because it doesn't require changes for
>> people who like to keep the patches applied in their VCS too. And
>> it's the principle of least surprise, you keep the same state afer a
>> build than you had before the build (so it's still ok for people who
>> rely on the scenario unpack/hack/rebuild).
>>
>> Comments? Does this look reasonable?
>
> I think that from 1.0 format, a use case is still broken with that
> approach:
> - build a package
> - install its binaries
> - check the behaviour is the same as with the package in the archive
> (i.e. rebuilding doesn't fix the issue magically)
> - hack hack hack
> - debuild|dpkg-buildpackage -b -nc
> ?? oh, files were patched/unpatched/changed, you get to rebuild a lot
> of things, instead of just the relevant files
>
> Unfortunately I come empty-handed, but I wanted to mention that use
> case anyway.
>
> Mraw,
> KiBi.

How is that? The default would still be that patches are applied, right?
So nothing would get unpatched in the default case.

For the case of having a VCS with patches unapplied you have conflicting
goals. You want the patches applied to keep rebuild minimal and
unapplied to be able to VCS commit without clean/unapply. You can't have
both. For that case I suggest using ccache. Or maybe a
"debuild|dpkg-buildpackage -b -nc --no-unapply"?

MfG Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 877h99y7c4.fsf@frosties.localnet">http://lists.debian.org/877h99y7c4.fsf@frosties.localnet
 
Old 05-29-2011, 01:51 PM
Goswin von Brederlow
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Raphael Hertzog <hertzog@debian.org> writes:

> Hello,
>
> from time to time I hear some rumblings about how "3.0 (quilt)" mixes
> badly with VCS. Indeed, one of the primary goals of the format
> was to not require prior knowledge of the patch system to be able to
> modify a package. And it's the case since you can do:
> - dpkg-source -x
> - modify files
> - debuild
>
> But there are a few downsides:
>
> Auto-application of patches
> ---------------------------
>
> Since dpkg-source -x auto-applies the patch, the build part was
> designed with the assumption that patches are pre-applied and this
> assumption is often wrong for people using VCS. I have resolved this
> by auto-applying the patch at the start of a build but (some) users of VCS
> have been annoyed with it because it left the VCS unclean after the
> build. The option --unapply-patches has been added as a way to change
> the behaviour (and you can put it debian/source/local-options).

I really like that patches are applied by dpkg-source -x by default and
I hope you will keep it that way. If one finds a buggy package and wants
to check the source it is enough to apt-get source the package and load
the files into an editor to see what the actualy code is.

If patches weren't applied then you would first have to find out how to
apply them, just like with the myriad of patch systems in 1.0 format.

> I would like to improve this situation and not force the majority of
> people to add the unapply-patches option (if it turns out the majority
> of people use this option or are annoyed by the patches being applied).
>
> I see 2 ways to solve this:
> a/ detect the common VCS and make --unapply-patches the default in that
> case (but it would require a --no-unapply-patches for the people who
> keep the patches applied in their VCS)

The problem isn't that people use a VCS but that their workflow is based
on not having patches applied. There is also another group of people
that work with patches applied. Specifically when you have feature
branches and those branches merged into master. In that workflow you can
even exclude debian/patches/ from the VCS and generate it from the
branches.

I don't think you can automatically detect the workflow being used so
this option is a no-go.

> b/ modify dpkg-source --before-build to keep a trace of the fact that
> it applied the patches (for example by creating
> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
> --after-build unapply the patches so that we're back to a clean
> state after a succesful build.
> If the build fails, we'd keep the patches applied.
>
> My preference goes to b/ because it doesn't require changes for people
> who like to keep the patches applied in their VCS too. And it's the
> principle of least surprise, you keep the same state afer a build than
> you had before the build (so it's still ok for people who rely
> on the scenario unpack/hack/rebuild).
>
> Comments? Does this look reasonable?

I would go one step further. Instead of recording wether you apply
patches or not record what patch the source is on before build. One
might be working on a patch halfway down the patch queue, run quilt
refresh, debuild -b -nc. That should return the working dir to the patch
it was before the build.

> Automatic patches
> -----------------
>
> Again to cope with the scenario explained at the start of this mail,
> once a user has made modifications we must ensure that they end up in a
> proper patch in debian/patches/. Right now this is entirely automatic,
> the generated patches take the form of
> debian/patches/debian-changes-<ver>.
>
> Obviously this is a pretty poor name for a patch and given it's
> automatically generated, it doesn't contain much useful description.
> In general they are renamed by the maintainer who also adds a
> description (or they are properly put in place before the build so that
> dpkg-source doesn't have to create it).
>
> But it still happens that those patches are generated[1] when the maintainer
> did not expect any change at all. That's why we added the option
> --abort-on-upstream-changes for maintainers who never wants dpkg-source
> to auto-create a patch.
>
> I wonder if I should not make this option the default and fail with an
> error messages suggesting that the maintainer should run "dpkg-source
> --record-changes" if he really wants to keep the current changes. It
> would also leave the diff around somewhere in $TMPDIR if the user wants
> to review it.
>
> It would break the initial scenario but since the solution is given in the
> error message, I believe it would be acceptable.
>
> "dpkg-source --record-changes <patch-name>" would be an interactive command
> because you would be asked to fill the patch description header.

(In case you have a tty Instead of giving an error why not ask for the
patch name and abort on empty name?

> We still have the case of people who keep all their changes in the VCS
> without maintaining debian/patches/ and who uses --single-debian-patch.
> In that case, --abort-on-upstream-changes would not apply.
>
> What do you think of this? Do you have suggestions to improve this
> workflow?

I have a suggestion to improve the case where debian/patches/ is not
kept in the VCS but is generated from it (having a single debian patch
being a special case of this).

I actualy have a slightly hackish script for this case that avoids
unpacking the orig.tar, applying all the patches and diffing. This can
greatly speed up generating a source package. The VCS ensures that what
the source package contains will match the working dir (assuming
branches-to-patches does it work right).

================================================== ====================
#!/bin/sh

set -ex

DIR="$(basename "$(pwd)")"
NAME=$(dpkg-parsechangelog | grep "^Source:" | cut -d" " -f2)
VERSION=$(dpkg-parsechangelog | grep "^Version:" | cut -d" " -f2)
UPSTREAM=${VERSION%-*}
FORMAT=$(dpkg-source --print-format .)

# Make sure we have 3.0 (quilt). No idea about 3.0 (git).
[ "$FORMAT" = "3.0 (quilt)" ] || { echo "Not a 3.0 (quilt) source."; exit 1; }

# Build debian/patches/*
# This creates a patch series from a series of branches by asking the
# VCS for a diff between them.
branches-to-patches debian/branches-to-patches.conf

# Build debian tarball
# This assumes you have a clean working dir, like when building with
# pdebuild in a chroot.
tar -czf "../${NAME}_${VERSION}.debian.tar.gz" debian

# Build dsc file
( cd .. &&
dpkg-source --format="3.0 (custom)" --target-format="$FORMAT" -b "$DIR" "${NAME}_${UPSTREAM}.orig.tar."* "${NAME}_${VERSION}.debian.tar.gz" )
================================================== ====================

The important part here is that I use branches-to-patches to update
debian/patches/*. It would be nice if one could specify something in
debian/source/local-options to run a command when debian/patches/*
should be updated instead of (or prior to) the default of generating
debian/patches/debian-changes[-version].

This would be usefull for the case of having patches applied in master
and not having them applied. That just depends on what
branches-to-patches does to generate the patches.

You don't have to trust that whatever command is specified does a proper
job. You can still do the untar+patch+diff dance to make sure. This
could still default to --abort-on-upstream-changes since any change
means the branches-to-patches did not create something that matches the
current working directory. The argument for/against defaulting to
--abort-on-upstream-changes apply here too or even more so for it.

Note: I specifically said local-options. This should not end up in the
generated source package, only in a VCS checkout.

> Ideally, if desired, dpkg-source --record-changes could be automatically
> called in the build process but I'm wary of introducing something
> interactive in the package build process.

At least have an option one can set to get this behaviour. I hate
programs that quit with "do xyz" just so I have to copy&paste that
command.

> Cheers,
>
> [1] http://lintian.debian.org/tags/format-3.0-but-debian-changes-patch.html

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 8739jxy5ri.fsf@frosties.localnet">http://lists.debian.org/8739jxy5ri.fsf@frosties.localnet
 
Old 05-30-2011, 02:38 PM
Raphael Hertzog
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

On Sun, 29 May 2011, Benjamin Drung wrote:
> Am Sonntag, den 29.05.2011, 10:53 +0200 schrieb Raphael Hertzog:
> > Again to cope with the scenario explained at the start of this mail,
> > once a user has made modifications we must ensure that they end up in a
> > proper patch in debian/patches/. Right now this is entirely automatic,
> > the generated patches take the form of
> > debian/patches/debian-changes-<ver>.
> >
> > Obviously this is a pretty poor name for a patch [...]
>
> The file should end with .patch
> (debian/patches/debian-changes-<ver>.patch) so that your favorite text
> editor uses the correct highlighting.

At the time I wrote it, it was on purpose that I did not use any
extension. It limited the possibilities of interaction if another patch
system was in use while still using "3.0 (quilt)".

In the mean time, I abandoned the idea of auto-migrating source packages
so it's probably no longer very important. Thus I could do that.

Are there objections to this change?

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110530143835.GC28898@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110530143835.GC28898@rivendell.home.ouaza.com
 
Old 05-30-2011, 02:41 PM
Raphael Hertzog
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

Hi,

On Sun, 29 May 2011, Bernhard R. Link wrote:
> That sounds a bit better, but it adds even more magic to dpkg-source.
> I really miss some way to express: "In this account, do not use magic.
> If things are not correct and need fixing, tell me what is wrong and
> abort so I'll never miss it." (Actually I'd prefer it as default and
> only have it enabled by some options, but a way to globally disable
> and turn them into hard errors would would be good enough[tm].)

I don't consider "applying/unapplying patches" as magic. :-)

> > But it still happens that those patches are generated[1] when the maintainer
> > did not expect any change at all. That's why we added the option
> > --abort-on-upstream-changes for maintainers who never wants dpkg-source
> > to auto-create a patch.
>
> There is still no way to get this behaviour account-wide, is there?

No. Except dpkg-buildflags none of the dpkg-dev scripts support
configuration files IIRC.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110530144155.GD28898@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110530144155.GD28898@rivendell.home.ouaza.com
 
Old 05-30-2011, 05:07 PM
gregor herrmann
 
Default Behaviour of dpkg-source with "3.0 (quilt)" and VCS and automatic patches

On Sun, 29 May 2011 10:53:03 +0200, Raphael Hertzog wrote:

> Auto-application of patches
> ---------------------------
>
> I would like to improve this situation and not force the majority of
> people to add the unapply-patches option (if it turns out the majority
> of people use this option or are annoyed by the patches being applied).

Yup, I was already considering to (propose to) add it (and also
abort-on-upstream-changes) to all pkg-perl packages to avoid checking
in patched files.

> b/ modify dpkg-source --before-build to keep a trace of the fact that
> it applied the patches (for example by creating
> .pc/dpkg-source-auto-applied) and in that case have dpkg-source
> --after-build unapply the patches so that we're back to a clean
> state after a succesful build.
> If the build fails, we'd keep the patches applied.

Sounds good to me.

> Automatic patches
> -----------------
>
> But it still happens that those patches are generated[1] when the maintainer
> did not expect any change at all. That's why we added the option
> --abort-on-upstream-changes for maintainers who never wants dpkg-source
> to auto-create a patch.
>
> I wonder if I should not make this option the default and fail with an
> error messages suggesting that the maintainer should run "dpkg-source
> --record-changes" if he really wants to keep the current changes. It
> would also leave the diff around somewhere in $TMPDIR if the user wants
> to review it.

Sounds also good to me.


Thanks for working on these improvements!


Cheers,
gregor

--
.'`. Homepage: http://info.comodo.priv.at/ - PGP/GPG key ID: 0x8649AA06
: :' : Debian GNU/Linux user, admin, & developer - http://www.debian.org/
`. `' Member of VIBE!AT & SPI, fellow of Free Software Foundation Europe
`- NP: Kurt Ostbahn & Die Kombo: Na, so wirst ned oid
 

Thread Tools




All times are GMT. The time now is 07:02 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org