FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-04-2009, 05:42 AM
Ben Finney
 
Default Pristine source from upstream VCS repository

Howdy all,

A Debian source package is constructed from a “pristine source”
tree, plus Debian-specific changes. The pristine source is usually the
tarball distributed by the upstream developer of the work.

The ‘uscan’ tool, as configured by the ‘debian/watch’ file in the
source package, allows assurance that every Debian user can get the
same pristine source, for as long as it's available from the same
upstream location — but only if it's available as a tarball.

Increasingly, some upstream developers are not making tarball releases
of their source, and Debian's “pristine source” is an exported
revision from an upstream version control system. The ‘uscan’ tool
currently has no way to accomodate these packages.


Debian's policy §4.9 discusses a ‘debian/rules’ target named
‘get-orig-source’:

`get-orig-source' (optional)
This target fetches the most recent version of the original
source package from a canonical archive site (via FTP or
WWW, for example), does any necessary rearrangement to turn
it into the original source tar file format described below,
and leaves it in the current directory.

I have had some success with the following approach:

* Name the Debian package's release version such that the upstream
version ends with the upstream VCS revision number, e.g.
‘foo-1.2.3~bzr.r987-1’ (the upstream version number is thus
‘1.2.3~bzr.r987’).

* Write a ‘debian/get-orig-source’ program that will:

* Parse the changelog to get the package name and upstream version.

* Parse the upstream version string to get the specified VCS
revision number.

* Specify the upstream VCS repository URL.

* Invoke the appropriate VCS tool to export the specified revision
from the VCS repository URL to a temporary directory.

* Pack the temporary directory to an appropriately-named tarball in
the current directory.

* In ‘debian/rules’, define ‘get-orig-source’ as a ‘.PHONY’ target,
and make its action run the program ‘debian/get-orig-source’.

* Write a ‘debian/watch’ that explains in comments that the upstream
source is fetched via ‘debian/rules get-orig-source’ as per policy.

For an example of this approach, see the ‘docutils-manpage-writer’
package. (The program ‘get-orig-source’ is licensed under GPLv2+,
share and enjoy.)


This gives satisfactory results, but it does require non-trivial
fiddling with each package's rules and programs to do the right thing.
I can't help thinking that it's only a stop-gap waiting for a true
solution.

I imagine a true solution would be to tell the Debian packaging tools
more directly that the source is to be fetched, not as a tarball, but
as a specified revision from a specified public VCS repository.

Is anyone working on something like this capability?

--
“It is hard to believe that a man is telling the truth when you |
` know that you would lie if you were in his place.” —Henry L. |
_o__) Mencken |
Ben Finney


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-04-2009, 05:51 AM
Ben Finney
 
Default Pristine source from upstream VCS repository

Ben Finney <ben+debian@benfinney.id.au> writes:

> For an example of this approach, see the ‘docutils-manpage-writer’
> package.

The package is actually named ‘docutils-writer-manpage’.

--
“The errors of great men are venerable because they are more |
` fruitful than the truths of little men.” —Friedrich Nietzsche |
_o__) |
Ben Finney


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-04-2009, 06:37 AM
Cyril Brulebois
 
Default Pristine source from upstream VCS repository

Ben Finney <ben+debian@benfinney.id.au> (04/03/2009):
> * Invoke the appropriate VCS tool to export the specified revision
> from the VCS repository URL to a temporary directory.
>
> * Pack the temporary directory to an appropriately-named tarball in
> the current directory.

AFAICT, that doesn't ensure reproducibility. See pristine-* for hints
about things that can change between one environment and another.

Mraw,
KiBi.
 
Old 03-04-2009, 07:09 AM
Ben Finney
 
Default Pristine source from upstream VCS repository

Cyril Brulebois <kibi@debian.org> writes:

> Ben Finney <ben+debian@benfinney.id.au> (04/03/2009):
> > * Invoke the appropriate VCS tool to export the specified revision
> > from the VCS repository URL to a temporary directory.
> >
> > * Pack the temporary directory to an appropriately-named tarball in
> > the current directory.
>
> AFAICT, that doesn't ensure reproducibility.

How feasible is it to ensure reproducibility, though, when there is no
canonical upstream release tarball?

> See pristine-* for hints about things that can change between one
> environment and another.

I don't know what ‘pristine-*’ refers to there. I get no result from
‘apropos pristine’, so if you're referring to a command, I have no
corresponding manpage.

--
“People demand freedom of speech to make up for the freedom of |
` thought which they avoid.” —Soren Aabye Kierkegaard (1813-1855) |
_o__) |
Ben Finney


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-04-2009, 07:16 AM
Paul Wise
 
Default Pristine source from upstream VCS repository

On Wed, Mar 4, 2009 at 5:09 PM, Ben Finney <ben+debian@benfinney.id.au> wrote:

>> See pristine-* for hints about things that can change between one
>> environment and another.
>
> I don't know what ‘pristine-*’ refers to there. I get no result from
> ‘apropos pristine’, so if you're referring to a command, I have no
> corresponding manpage.

See pristine-tar and the commands therein.

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-04-2009, 07:19 AM
Stefano Zacchiroli
 
Default Pristine source from upstream VCS repository

On Wed, Mar 04, 2009 at 07:09:07PM +1100, Ben Finney wrote:
> I don't know what ‘pristine-*’ refers to there. I get no result from
> ‘apropos pristine’, so if you're referring to a command, I have no
> corresponding manpage.

See the pristine-tar package.

--
Stefano Zacchiroli -o- PhD in Computer Science PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu * tous ceux que j'aime
 
Old 03-04-2009, 07:19 AM
Cyril Brulebois
 
Default Pristine source from upstream VCS repository

Ben Finney <ben+debian@benfinney.id.au> (04/03/2009):
> How feasible is it to ensure reproducibility, though, when there is no
> canonical upstream release tarball?

Just saying that your get-orig-source stuff will not necessarily give
you the same tarball on different machines/setups/etc.

You want to look at pristine-tar to look at what can be done to do so.

> > See pristine-* for hints about things that can change between one
> > environment and another.
>
> I don't know what ‘pristine-*’ refers to there. I get no result from
> ‘apropos pristine’, so if you're referring to a command, I have no
> corresponding manpage.

| $ apt-cache search pristine
| dbs - Allows Debian source packages with multiple patches
| linux-patch-debian-2.6.28 - Debian patches to version 2.6.28 of the Linux kernel
| pristine-tar - regenerate pristine tarballs

See the last one. pristine-{gz,bz2} in particular.

Mraw,
KiBi.
 
Old 03-04-2009, 09:08 AM
Ben Finney
 
Default Pristine source from upstream VCS repository

Cyril Brulebois <kibi@debian.org> writes:

> | $ apt-cache search pristine
> | dbs - Allows Debian source packages with multiple patches
> | linux-patch-debian-2.6.28 - Debian patches to version 2.6.28 of the Linux kernel
> | pristine-tar - regenerate pristine tarballs
>
> See the last one. pristine-{gz,bz2} in particular.

Interesting.

I'm not sure how to actually use it in a Debian package though.

Reading the manual pages, I get the impression it's designed to
produce deltas that upstream then commits in their VCS. If I don't
have access to commit to the upstream VCS, how can I make use of this?

--
“I spilled spot remover on my dog. Now he's gone.” —Steven |
` Wright |
_o__) |
Ben Finney


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-04-2009, 09:25 AM
Stefano Zacchiroli
 
Default Pristine source from upstream VCS repository

On Wed, Mar 04, 2009 at 09:08:08PM +1100, Ben Finney wrote:
> I'm not sure how to actually use it in a Debian package though.
>
> Reading the manual pages, I get the impression it's designed to
> produce deltas that upstream then commits in their VCS. If I don't
> have access to commit to the upstream VCS, how can I make use of
> this?

It is in fact mainly used in Debian to store efficiently tarballs
within VCSs, tarball usually released upstream though. The problem
here is that to have checksum-safe upstream tarball, you will indeed
need to convince upstream to use stuff like pristine-tar. I believe
that in general, it will be quit a hard task, as upstream are not
concerned with the problem as much as we are.

What you can do is probably have a 2 level architecture, with a VCS
Debian-side which mirrors upstream VCS (easy with DVCS) but has in
addition the deltas. The first time your tarball fetching tool will
create the tarball and store its delta. The other times it will just
use pristine-tar to check it out.

... but it starts looking like a bit hackish.

Cheers.

--
Stefano Zacchiroli -o- PhD in Computer Science PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c' ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu tous ceux que j'aime
 
Old 03-04-2009, 09:30 AM
Adeodato Sim
 
Default Pristine source from upstream VCS repository

* Stefano Zacchiroli [Wed, 04 Mar 2009 11:25:23 +0100]:

> What you can do is probably have a 2 level architecture, with a VCS
> Debian-side which mirrors upstream VCS (easy with DVCS) but has in
> addition the deltas. The first time your tarball fetching tool will
> create the tarball and store its delta. The other times it will just
> use pristine-tar to check it out.

I haven't used it in this way, but one should investigate if the need
for the mirror VCS is needed at all.

From the first paragraph in the manpage, it would seem that pristine-tar
just wants a "checkout" of the appropriate revision, i.e. an unpacked
tree. So you could checkout that from the upstream VCS, and store the
delta in the debian-only VCS, if in fact your packaging VCS is debian-only.
(Else, this point is moot. :-)

I hope I explained myself clearly.

Cheers,

--
Adeodato Sim dato at net.com.org.es
Debian Developer adeodato at debian.org

As scarce as truth is, the supply has always been in excess of the demand.
-- Josh Billings


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 06:00 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org