FAQ Search Today's Posts Mark Forums Read

» Linux Archive
Home
New Posts
Search
FAQ


Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 02-06-2010, 04:30 PM
Jonathan Nieder
 
Default dpkg source format 3.0 (git)

Henrique de Moraes Holschuh wrote:

> Given the size of the .git dir when you're not using something like stgit
> and have all objects properly packed, one could even consider shipping .git
> inside the package if it is small enough.
>
> Is that a proper way to do this? Or would we need a new 3.0 format variant
> to do it right (assuming we want to do that) ?

There has been talk [1] of a format variant that uses git bundles to do
that efficiently, but nothing concrete. To me, it seems like a good idea.

A few points [2] to keep in mind:

- People can easily download source packages to access the original
source and Debian changes without too much technical
sophistication. Even a quilt series is just a series of patches so
it doesn’t require specialized tools. So there is precedent for
making it easy to access a source package from Debian.

Will ‘git’ ever be as ubiquitous as ‘patch’? If the source package
is something you can ‘git clone’ to get a repository with
appropriate ‘master’ and ‘origin/upstream’ branches, is that good
enough? Probably. ;-)

- The bundle needs to be clonable without use of any external object
repository. The bundle ought to have branches ‘master’ and
‘upstream’, relevant tags, maybe a ‘pristine-tar’ branch, and not
much extra junk.

- To be included in Debian main, everything in the bundle should meet
the free software guidelines (right?); practically speaking, this
means the history would very often be cauterized when the project
is first packaged for Debian.

Maybe some of the damage from this could be mitigated by including
a suitable grafts file or refs for use by ‘git replace’.

- Including the whole history with each upload and forcing users and
buildds to reclone with each download would be unpleasant. (This
can be turned around: with work, a git-based format might be able
to decrease bandwidth use for sites like snapshots.debian.org.)

- Bundles are hard to recreate. If they were to be created on the
fly some way as in the previous item, the md5sums are not stable.
Just something to think about.

Thoughts? Especially, if anyone would like to talk implementation,
let me know. It sounds difficult to do right, but there’s no harm in
experimenting.

The easiest way to try this out now is to include a bundle in the
source tarball (ugh) and clone it at build time. If you’re doing
that, please let me know how it goes.

Jonathan

[1] http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11234
[2] Thanks to Goswin for many of these:
http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11234/focus=11264


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-09-2010, 11:42 AM
Goswin von Brederlow
 
Default dpkg source format 3.0 (git)

Jonathan Nieder <jrnieder@gmail.com> writes:

> Henrique de Moraes Holschuh wrote:
>
>> Given the size of the .git dir when you're not using something like stgit
>> and have all objects properly packed, one could even consider shipping .git
>> inside the package if it is small enough.
>>
>> Is that a proper way to do this? Or would we need a new 3.0 format variant
>> to do it right (assuming we want to do that) ?
>
> There has been talk [1] of a format variant that uses git bundles to do
> that efficiently, but nothing concrete. To me, it seems like a good idea.
>
> A few points [2] to keep in mind:
>
> - People can easily download source packages to access the original
> source and Debian changes without too much technical
> sophistication. Even a quilt series is just a series of patches so
> it doesnâ??t require specialized tools. So there is precedent for
> making it easy to access a source package from Debian.
>
> Will â??gitâ?? ever be as ubiquitous as â??patchâ??? If the source package
> is something you can â??git cloneâ?? to get a repository with
> appropriate â??masterâ?? and â??origin/upstreamâ?? branches, is that good
> enough? Probably. ;-)
>
> - The bundle needs to be clonable without use of any external object
> repository. The bundle ought to have branches â??masterâ?? and
> â??upstreamâ??, relevant tags, maybe a â??pristine-tarâ?? branch, and not
> much extra junk.
>
> - To be included in Debian main, everything in the bundle should meet
> the free software guidelines (right?); practically speaking, this
> means the history would very often be cauterized when the project
> is first packaged for Debian.
>
> Maybe some of the damage from this could be mitigated by including
> a suitable grafts file or refs for use by â??git replaceâ??.
>
> - Including the whole history with each upload and forcing users and
> buildds to reclone with each download would be unpleasant. (This
> can be turned around: with work, a git-based format might be able
> to decrease bandwidth use for sites like snapshots.debian.org.)
>
> - Bundles are hard to recreate. If they were to be created on the
> fly some way as in the previous item, the md5sums are not stable.
> Just something to think about.
>
> Thoughts? Especially, if anyone would like to talk implementation,
> let me know. It sounds difficult to do right, but thereâ??s no harm in
> experimenting.

Two things:

1) Think about doing this for linux-2.6, XOrg or OOo and what it would
mean for the source size or usability.

2) If you want a git repository then use a git repository. Castrating
git, hacking the history to pices and then mangling the whole thing into
a tar.gz is just stupid.

Teach DAK, dpkg-source and libapt about git. The Sources.gz file would
then just contain a reference and apt-get source foo would clone the
repository and so on.

Uploading a new source could then be sending a signed ref to the
maintainers git or sending a signed bundle or even just pushing and
setting a tag.

> The easiest way to try this out now is to include a bundle in the
> source tarball (ugh) and clone it at build time. If youâ??re doing
> that, please let me know how it goes.

But not the best way.

> Jonathan
>
> [1] http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11234
> [2] Thanks to Goswin for many of these:
> http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11234/focus=11264

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-09-2010, 05:03 PM
Jonathan Nieder
 
Default dpkg source format 3.0 (git)

Goswin von Brederlow wrote:

> 1) Think about doing this for linux-2.6, XOrg or OOo and what it would
> mean for the source size or usability.

Indeed, the history is pretty large. A rough estimate (for something
less rough, one should use a well packed bundle with only the objects
of interest):

$ pwd
/home/jrn/src/linux-2.6
$ git status -u
# On branch next
nothing to commit (working directory clean)
$ du -sk .git
440664 .git
$ du -sk --exclude=.git .
450920 .
$ du -sk ../linux-2.6.33-rc7.tar.bz2
64648 ../linux-2.6.33-rc7.tar.bz2

The source package would become about 7 times as large.

For usability: I imagine what is typically needed is the set of Vcs-Git
fields somewhere conveniently machine-readable, so one could just go

apt-get source --git whatever

and get a checkout of its packaging repository.

That would be the 90% solution. What it doesn’t do is help people with
poor connectivity to hack on a package like linux-2.6. Given the high
quality of commit messages and the sparsity of comments on the
implementation, it is really much easier to work with the history in
hand.

So it would be nice (though hard) to find some method that allows the
git history to be widely mirrored, included on distributed DVDs, and
so on. I’m sure the admins of kernel.org would appreciate it as well.

> Uploading a new source could then be sending a signed ref to the
> maintainers git or sending a signed bundle or even just pushing and
> setting a tag.

I imagine that would be very nice for people with large packages.
Maybe something similar could be accomplished for existing
tarball+changes packages by providing a "proxy" git server that runs
dpkg-source -b server side.

Selfishly, I guess if someone implements the 90% solution above, I
would stop caring so much about what source format the buildds use.
Others might be more principled, though. ;-)

Regards,
Jonathan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-10-2010, 06:36 AM
Goswin von Brederlow
 
Default dpkg source format 3.0 (git)

Jonathan Nieder <jrnieder@gmail.com> writes:

> Goswin von Brederlow wrote:
>
>> 1) Think about doing this for linux-2.6, XOrg or OOo and what it would
>> mean for the source size or usability.
>
> Indeed, the history is pretty large. A rough estimate (for something
> less rough, one should use a well packed bundle with only the objects
> of interest):
>
> $ pwd
> /home/jrn/src/linux-2.6
> $ git status -u
> # On branch next
> nothing to commit (working directory clean)
> $ du -sk .git
> 440664 .git
> $ du -sk --exclude=.git .
> 450920 .
> $ du -sk ../linux-2.6.33-rc7.tar.bz2
> 64648 ../linux-2.6.33-rc7.tar.bz2
>
> The source package would become about 7 times as large.
>
> For usability: I imagine what is typically needed is the set of Vcs-Git
> fields somewhere conveniently machine-readable, so one could just go
>
> apt-get source --git whatever
>
> and get a checkout of its packaging repository.
>
> That would be the 90% solution. What it doesnâ??t do is help people with
> poor connectivity to hack on a package like linux-2.6. Given the high
> quality of commit messages and the sparsity of comments on the
> implementation, it is really much easier to work with the history in
> hand.

Then create CD/DVDs of git.d.o.

> So it would be nice (though hard) to find some method that allows the
> git history to be widely mirrored, included on distributed DVDs, and
> so on. Iâ??m sure the admins of kernel.org would appreciate it as
> well.

Show the demand and I'm sure mirroring will follow.

>> Uploading a new source could then be sending a signed ref to the
>> maintainers git or sending a signed bundle or even just pushing and
>> setting a tag.
>
> I imagine that would be very nice for people with large packages.
> Maybe something similar could be accomplished for existing
> tarball+changes packages by providing a "proxy" git server that runs
> dpkg-source -b server side.

For non-upstream updates the upload is usualy small. Just the diff.gz or
debian.tar.gz file. But yeah, for most upstream changes a pristine-tar
upload is much smaller than the full orig.tar.gz. But is that really the
problem?

> Selfishly, I guess if someone implements the 90% solution above, I
> would stop caring so much about what source format the buildds use.
> Others might be more principled, though. ;-)
>
> Regards,
> Jonathan

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2010, 08:58 PM
Jonathan Nieder
 
Default dpkg source format 3.0 (git)

Jonathan Nieder wrote:

> For usability: I imagine what is typically needed is the set of Vcs-Git
> fields somewhere conveniently machine-readable, so one could just go
>
> apt-get source --git whatever
>
> and get a checkout of its packaging repository.

... which is what debcheckout provides, though I didn’t know it. Thanks
to Stefano Zacchiroli for writing it!

> Selfishly, I guess if someone implements the 90% solution above, I
> would stop caring so much about what source format the buildds use.
> Others might be more principled, though. ;-)

So indeed, I do not think git-based distributed source packages are
worth worrying about yet. In those places where git.debian.org and co
are not good enough, I suspect it would be a better use of time to try
to improve connectivity.

Jonathan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100216215855.GA3184@progeny.tock">http://lists.debian.org/20100216215855.GA3184@progeny.tock
 

Thread Tools




All times are GMT. The time now is 09:29 AM.

VBulletin, Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org