Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On Fri, Aug 17, 2012 at 12:26:49PM +0200, Mike Hommey wrote:
> > > > As an unrelated idea which popped up when reading this: Do you think it
> > > > would be a sensible enhancement to uupdate if it could deal with a list
> > > > of files (wildcard strings that could be feed to `rm -rf`) which should
> > > > be removed from the upstream tarball? This could simplify repackaging
> > > > to a certain amount.
> > >
> > > I use a script for this:
> > >
> > > http://anonscm.debian.org/viewvc/debichem/unstable/openbabel/debian/watch?view=markup
> > > http://anonscm.debian.org/viewvc/debichem/unstable/openbabel/debian/get-orig-source.sh?view=markup
> > >
> > > There are several other examples in this tree. It is pretty simple. Just
> > > replace line 20 in get-orig-source.sh with whatever you want to do here.
> > > FYI: I don't make use of uupdate. I don't need it in my setup.
Sure, you need to replace uupdate by your own script and my suggestion
was to let uupdate do this for you by a unified interface instead of
having everybody writing its own debian/get-orig-source script.
> > I'm using a more sophisticated script, that allows to filter at the same
> > time as the file is downloaded, without actually extracting to disk.
> > http://anonscm.debian.org/gitweb/?p=pkg-mozilla/iceweasel.git;a=blob;f=debian/repack.py;h=a797d5471f20e0f8de155d483e5ad2f1b2c3bd c5;hb=c1ebf8be93add288837377e4fdd87f9c9f1082cc
> >
> > This can seem a worthless optimization, but when your source tarball is
> > more than 80MB bzipped, it makes a huge difference (especially when
> > downloading is rather slow, in which case the whole process is about as
> > fast as downloading alone)
> >
> > It supports wildcards, and extra filters (sed-like). See, for example,
> > the filter used on iceweasel source:
> > http://anonscm.debian.org/gitweb/?p=pkg-mozilla/iceweasel.git;a=blob;f=debian/source.filter;h=ec7efac7b97add1f39480c07fecb4b70ae 7a7ec8;hb=c1ebf8be93add288837377e4fdd87f9c9f1082cc
>
> Oh, and the script just works out of the box with uscan:
>
> http://anonscm.debian.org/gitweb/?p=pkg-mozilla/iceweasel.git;a=blob;f=debian/watch;h=a81586cc891ab608c52717069c99703227e4d077;h b=c1ebf8be93add288837377e4fdd87f9c9f1082cc
So we finally have three independently developed solutions (we also have
several instances of a debian/get-orig-source script in Debian Med
team) and my suggestion was just to settle with a common and simple
solution. This should be pretty simple to implement (I'd volunteer to
do this but wanted to seek for comments before filing a bug report +
patch).
Kind regards
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120817110839.GA6417@an3as.eu">http://lists.debian.org/20120817110839.GA6417@an3as.eu
08-17-2012, 11:47 AM
gregor herrmann
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On Fri, 17 Aug 2012 13:08:39 +0200, Andreas Tille wrote:
> So we finally have three independently developed solutions (we also have
> several instances of a debian/get-orig-source script in Debian Med
> team) and
pkg-perl variant:
http://anonscm.debian.org/gitweb/?p=pkg-perl/scripts.git;a=tree
repack.sh and repack.stub there
> my suggestion was just to settle with a common and simple
> solution. This should be pretty simple to implement (I'd volunteer to
> do this but wanted to seek for comments before filing a bug report +
> patch).
Yup, that would be nice.
But I don't think it belongs into uupdate but in uscan.
Cheers,
gregor
--
.'`. Homepage: http://info.comodo.priv.at/ - OpenPGP key 0xBB3A68018649AA06
: :' : Debian GNU/Linux user, admin, and developer - http://www.debian.org/
`. `' Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
`- NP: Stevie Wonder: Your Are the Sunshine of My Life
08-17-2012, 12:12 PM
Andreas Tille
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
Hi Gregor,
On Fri, Aug 17, 2012 at 01:47:48PM +0200, gregor herrmann wrote:
> On Fri, 17 Aug 2012 13:08:39 +0200, Andreas Tille wrote:
>
> > So we finally have three independently developed solutions (we also have
> > several instances of a debian/get-orig-source script in Debian Med
> > team) and
>
> pkg-perl variant:
> http://anonscm.debian.org/gitweb/?p=pkg-perl/scripts.git;a=tree
> repack.sh and repack.stub there
Solution 4. ;-)
> > my suggestion was just to settle with a common and simple
> > solution. This should be pretty simple to implement (I'd volunteer to
> > do this but wanted to seek for comments before filing a bug report +
> > patch).
>
> Yup, that would be nice.
> But I don't think it belongs into uupdate but in uscan.
Right that's correct (and explains Daniel's hint there is no need for
uupdate).
Discussion:
1. I propose a file debian/uscan.remove which contains regular expressions.
The deletion process will loop over every line and does
rm -rf ${MAIN_SOURCE_DIR}/<expression>
2. uscan just knows the option --repack. Do we want to reuse this and start
deleting files in case the file debian/uscan.remove exists or do we
want to use an explicite option --repack-and-remove? I have some
preference for just using --repack.
3. If the tarball did not contained any of the strings in
debian/uscan.remove it should be left untouched (except if the repackaging
is needed because of compression method anyway)
4. In case something was removed the version string will be appended by
'~dfsg' to express the fact that the content of the original source
was changed.
What do you think?
Kind regards
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120817121249.GE6417@an3as.eu">http://lists.debian.org/20120817121249.GE6417@an3as.eu
08-17-2012, 02:38 PM
Jonas Smedegaard
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On 12-08-17 at 02:12pm, Andreas Tille wrote:
> Hi Gregor,
>
> On Fri, Aug 17, 2012 at 01:47:48PM +0200, gregor herrmann wrote:
> > On Fri, 17 Aug 2012 13:08:39 +0200, Andreas Tille wrote:
> >
> > > So we finally have three independently developed solutions (we also have
> > > several instances of a debian/get-orig-source script in Debian Med
> > > team) and
> >
> > pkg-perl variant:
> > http://anonscm.debian.org/gitweb/?p=pkg-perl/scripts.git;a=tree
> > repack.sh and repack.stub there
>
> Solution 4. ;-)
>
> > > my suggestion was just to settle with a common and simple
> > > solution. This should be pretty simple to implement (I'd volunteer to
> > > do this but wanted to seek for comments before filing a bug report +
> > > patch).
> >
> > Yup, that would be nice.
> > But I don't think it belongs into uupdate but in uscan.
>
> Right that's correct (and explains Daniel's hint there is no need for
> uupdate).
>
> Discussion:
>
> 1. I propose a file debian/uscan.remove which contains regular expressions.
> The deletion process will loop over every line and does
>
> rm -rf ${MAIN_SOURCE_DIR}/<expression>
It is already required (or only recommended?) to document in
debian/copyright when source has been altered, and for that file we
already have a well defined format for listing file content as part of
Copyright file format 1.0, a.k.a. DEP5.
I suggest instead of adding a new file to instead check if
debian/copyright contains as first line a supported URL (i.e. currently
http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
is the only valid URL), and if so parses the header paragraph for the
undocumented-but-also-not-disallowed field "Files-Excluded", like this:
[x] quote me freely [ ] ask before reusing [ ] keep private
08-18-2012, 08:19 PM
Andreas Tille
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
Hi,
trying to summarise suggested changes for the proposal:
1. The new field Files-Excluded in debian/copyright contains
a space separated list of regular expressions.
The deletion process will loop over every expression
2. There was one vote from Gregor Hermann to use the --repack option
of uscan. I personally admit that I do not fully agree with
Gregor that this means changing the semantics of an existing option.
We are just repackaging and without the additional information in
debian/copyright the functionality remains exactly the same.
If we would follow Bastien ROUCARIES suggestion to provide an
option for the compression method *this* would change the semantics
effectively and so I'm a bit reluctand to implement this in this
connection (even if I think it makes perfectly sense.
Gregor made also the remark that he likes the pkg-perl teams
method of doing things because once d/watch is adjusted you don't
have to do anything besides calling uscan. I consider the method
I like to suggest here as serving the very same purpose
3. If the tarball did not contained any of the strings in
debian/copyright::Files-Excluded it should be left untouched
(except if the repackaging is needed because of compression method
anyway)
4. In case something was removed the version string will be appended by
'+dfsg' to express the fact that the content of the original source
was changed.
For the implementation of this it waqs suggested to
use Debian::Copyright;
Any further hints / remarks?
Kind regards
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120818201922.GA12492@an3as.eu">http://lists.debian.org/20120818201922.GA12492@an3as.eu
08-18-2012, 10:44 PM
Jonas Smedegaard
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On 12-08-18 at 10:19pm, Andreas Tille wrote:
> 1. The new field Files-Excluded in debian/copyright contains
> a space separated list of regular expressions.
> The deletion process will loop over every expression
>
> rm -rf ${MAIN_SOURCE_DIR}/<expression>
Copyright file format emplicitly emphasizes that the globbing is not
shell style but find style.
I believe it is better to loop over either of these expressions:
find ${MAIN_SOURCE_DIR}/* type f -name <expression> -delete
The latter is when item does not contain "/".
> 4. In case something was removed the version string will be appended by
> '+dfsg' to express the fact that the content of the original source
> was changed.
Suffix should be configurable.
I use ~dfsg by default, ~dfsg1 and bumping numbers for multiple
repackagings, and only +dfsg when the repackaging happens after a
non-repackaged version was released into Debian.
Reason for this is that there is a slight chance upstream may re-release
same upstream version repackaged to fix a purely tarball-related issuem
and I would then have room for using that proper version instead of
using epoch or add a bogus .0 to the version.
> For the implementation of this it waqs suggested to
>
> use Debian::Copyright;
That initial test by Gregor makes me worry if Debian::Copyright parser
might be too strict: Writing should be strict but parsing relaxed -
Copyright file format with undefined fields added should *not* be
treated as broken. Perhaps there are other surprises waiting to happen
:-/
[x] quote me freely [ ] ask before reusing [ ] keep private
08-18-2012, 11:32 PM
Charles Plessy
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
Le Sun, Aug 19, 2012 at 12:44:44AM +0200, Jonas Smedegaard a écrit :
> On 12-08-18 at 10:19pm, Andreas Tille wrote:
> > 1. The new field Files-Excluded in debian/copyright contains
> > a space separated list of regular expressions.
> > The deletion process will loop over every expression
> >
> > rm -rf ${MAIN_SOURCE_DIR}/<expression>
>
> Copyright file format emplicitly emphasizes that the globbing is not
> shell style but find style.
>
> I believe it is better to loop over either of these expressions:
>
> find ${MAIN_SOURCE_DIR}/* -path <expression> -delete
>
> find ${MAIN_SOURCE_DIR}/* type f -name <expression> -delete
Hi all,
it looks like it is necessary to add './' before the expression if (and only
if) the expression does not contain it.
Have a nice Sunday,
--
Charles Plessy
Tsurumi, Kanagawa, Japan
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120818233239.GC20285@falafel.plessy.net">http://lists.debian.org/20120818233239.GC20285@falafel.plessy.net
08-19-2012, 08:59 AM
Jonas Smedegaard
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On 12-08-19 at 08:32am, Charles Plessy wrote:
> Le Sun, Aug 19, 2012 at 12:44:44AM +0200, Jonas Smedegaard a écrit :
> > On 12-08-18 at 10:19pm, Andreas Tille wrote:
> > > 1. The new field Files-Excluded in debian/copyright contains
> > > a space separated list of regular expressions.
> > > The deletion process will loop over every expression
> > >
> > > rm -rf ${MAIN_SOURCE_DIR}/<expression>
> >
> > Copyright file format emplicitly emphasizes that the globbing is not
> > shell style but find style.
> >
> > I believe it is better to loop over either of these expressions:
> >
> > find ${MAIN_SOURCE_DIR}/* -path <expression> -delete
> >
> > find ${MAIN_SOURCE_DIR}/* type f -name <expression> -delete
>
> Hi all,
>
> it looks like it is necessary to add './' before the expression if
> (and only if) the expression does not contain it.
Isn't that covered by the inverse above - trailing /* to base path?
[x] quote me freely [ ] ask before reusing [ ] keep private
08-19-2012, 10:19 AM
Andreas Tille
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
On Sun, Aug 19, 2012 at 12:44:44AM +0200, Jonas Smedegaard wrote:
> On 12-08-18 at 10:19pm, Andreas Tille wrote:
> > 1. The new field Files-Excluded in debian/copyright contains
> > a space separated list of regular expressions.
> > The deletion process will loop over every expression
> >
> > rm -rf ${MAIN_SOURCE_DIR}/<expression>
>
> Copyright file format emplicitly emphasizes that the globbing is not
> shell style but find style.
>
> I believe it is better to loop over either of these expressions:
>
> find ${MAIN_SOURCE_DIR}/* -path <expression> -delete
>
> find ${MAIN_SOURCE_DIR}/* type f -name <expression> -delete
>
> The latter is when item does not contain "/".
ACK.
> > 4. In case something was removed the version string will be appended by
> > '+dfsg' to express the fact that the content of the original source
> > was changed.
>
> Suffix should be configurable.
>
> I use ~dfsg by default, ~dfsg1 and bumping numbers for multiple
> repackagings, and only +dfsg when the repackaging happens after a
> non-repackaged version was released into Debian.
>
> Reason for this is that there is a slight chance upstream may re-release
> same upstream version repackaged to fix a purely tarball-related issuem
> and I would then have room for using that proper version instead of
> using epoch or add a bogus .0 to the version.
This was also my initial idea when firt proposing ~dfsg. On the other
hand: I would *really* want to have upstream adding a new version number
to the cleaned up release. It is just (uhmm, find your own word here)
if people release the same named file with different content. So I do
not see great harm if we would settle with +dfsg. Gregor, could you give
better reasons than Jonas for +dfsg? I'm personally a friend of adding
one feature (the removal of files) first. Later we could add additional
features like configuring suffixes and compression methods.
> > For the implementation of this it waqs suggested to
> >
> > use Debian::Copyright;
>
> That initial test by Gregor makes me worry if Debian::Copyright parser
> might be too strict: Writing should be strict but parsing relaxed -
> Copyright file format with undefined fields added should *not* be
> treated as broken. Perhaps there are other surprises waiting to happen
> :-/
Could anybody say something about this?
Kind regards
Andreas.
--
http://fam-tille.de
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120819101957.GC12722@an3as.eu">http://lists.debian.org/20120819101957.GC12722@an3as.eu
08-24-2012, 12:37 PM
Ian Jackson
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
Andreas Tille writes ("Re: Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)"):
> 1. The new field Files-Excluded in debian/copyright contains
> a space separated list of regular expressions.
> The deletion process will loop over every expression
>
> rm -rf ${MAIN_SOURCE_DIR}/<expression>
I don't think debian/copyright should be an controlling input to the
build process. It's supposed to be semi-machine-readable
documentation.
Ian.
--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20535.30078.862723.84291@chiark.greenend.org.uk">h ttp://lists.debian.org/20535.30078.862723.84291@chiark.greenend.org.uk
Enabling uupdate to simply remove files from upstream source (Was: Minified javascript files)
