FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Portage Developer

 
 
LinkBack Thread Tools
 
Old 06-01-2010, 03:32 AM
Zac Medico
 
Default Package compression header for binhosts

Hi,

In order to support alternative compression types for binhost
packages, I was thinking about adding support for a header field in
the Packages index file. For example, a header line like
"PACKAGE_EXTENSION: txz" could be used to indicate that clients
should download files with txz extensions instead of tbz2
extensions. I'm planning to add support for both tgz [1] and txz
extensions.

[1] http://bugs.gentoo.org/show_bug.cgi?id=142579
--
Thanks,
Zac
 
Old 06-01-2010, 05:16 AM
Brian Harring
 
Default Package compression header for binhosts

On Mon, May 31, 2010 at 08:32:34PM -0700, Zac Medico wrote:
> Hi,
>
> In order to support alternative compression types for binhost
> packages, I was thinking about adding support for a header field in
> the Packages index file. For example, a header line like
> "PACKAGE_EXTENSION: txz" could be used to indicate that clients
> should download files with txz extensions instead of tbz2
> extensions. I'm planning to add support for both tgz [1] and txz
> extensions.
>
> [1] http://bugs.gentoo.org/show_bug.cgi?id=142579

1) requires a version header bump
2) a header alone isn't useful unless it's specifiable per cpv entry;
thus it must be inheritable
3) PACKAGE_EXTENSION is overly verbose and unclear it's specifying
the compressor too; it's intention is for compression, state it as
such (I mention this in light of URI's existance where
PACKAGE_EXTENSION would only be a hint of compressor)

Re: #1, there is a decent set of optimizations I'm kicking around in
pkgcore for the next version- a discussion should probably be started
there.

Offhand, having a compression specific header (a simple enumeration
of known compressors) and a DEFAULT_URI that is python string
interpolation assembled (for example,
DEFAULT_URI="%(host)s/%(category)s/%(pf)s.txz") seems wiser. Via
doing what I'm suggesting, it would be possible to do binpkg
repository 'views' w/out having to map each binpkg into the url space
for it.

~harring
 
Old 06-01-2010, 08:01 PM
Ned Ludd
 
Default Package compression header for binhosts

On Mon, 2010-05-31 at 22:16 -0700, Brian Harring wrote:
> On Mon, May 31, 2010 at 08:32:34PM -0700, Zac Medico wrote:
> > Hi,
> >
> > In order to support alternative compression types for binhost
> > packages, I was thinking about adding support for a header field in
> > the Packages index file. For example, a header line like
> > "PACKAGE_EXTENSION: txz" could be used to indicate that clients
> > should download files with txz extensions instead of tbz2
> > extensions. I'm planning to add support for both tgz [1] and txz
> > extensions.
> >
> > [1] http://bugs.gentoo.org/show_bug.cgi?id=142579
>
> 1) requires a version header bump

Agreed. But there were some other pending changes for "VERSION: 1"

Any planned changes to the format should be documented on
https://bugs.gentoo.org/show_bug.cgi?id=263994


> 2) a header alone isn't useful unless it's specifiable per cpv entry;
> thus it must be inheritable

Per CPV entries is going to bloat the format and make me carry around a
more data on a per pkg basis then I'd want to. How about we run with
zac's idea but use tools to convert a full repo over to $EXTENTION
This should keep the portage code fast as well as it checks for invalid
binpkgs all the time. Having to have portage process a ton of ever
growing extentions is just going to be slow.

> 3) PACKAGE_EXTENSION is overly verbose and unclear it's specifying
> the compressor too; it's intention is for compression, state it as
> such (I mention this in light of URI's existance where
> PACKAGE_EXTENSION would only be a hint of compressor)
>
> Re: #1, there is a decent set of optimizations I'm kicking around in
> pkgcore for the next version- a discussion should probably be started
> there.
>
> Offhand, having a compression specific header (a simple enumeration
> of known compressors) and a DEFAULT_URI that is python string

No go bro. The 'Packages' format should be independent of python.

> interpolation assembled (for example,
> DEFAULT_URI="%(host)s/%(category)s/%(pf)s.txz") seems wiser. Via
> doing what I'm suggesting, it would be possible to do binpkg
> repository 'views' w/out having to map each binpkg into the url space
> for it.
>
> ~harring

--
Ned Ludd <solar@gentoo.org>
Gentoo Linux
 
Old 06-01-2010, 09:22 PM
Brian Harring
 
Default Package compression header for binhosts

On Tue, Jun 1, 2010 at 1:01 PM, Ned Ludd <solar@gentoo.org> wrote:


On Mon, 2010-05-31 at 22:16 -0700, Brian Harring wrote:

> On Mon, May 31, 2010 at 08:32:34PM -0700, Zac Medico wrote:

> > Hi,

> >

> > In order to support alternative compression types for binhost

> > packages, I was thinking about adding support for a header field in

> > the Packages index file. For example, a header line like

> > "PACKAGE_EXTENSION: txz" could be used to indicate that clients

> > should download files with txz extensions instead of tbz2

> > extensions. I'm planning to add support for both tgz [1] and txz

> > extensions.

> >

> > [1] http://bugs.gentoo.org/show_bug.cgi?id=142579

>

> 1) requires a version header bump



Agreed. But there were some other pending changes for "VERSION: 1"



Any planned changes to the format should be documented on

https://bugs.gentoo.org/show_bug.cgi?id=263994





> 2) a header alone isn't useful unless it's specifiable per cpv entry;

> thus it must be inheritable



Per CPV entries is going to bloat the format and make me carry around a

more data on a per pkg basis then I'd want to. How about we run with

zac's idea but use tools to convert a full repo over to $EXTENTION

This should keep the portage code fast as well as it checks for invalid

binpkgs all the time. Having to have portage process a ton of ever

growing extentions is just going to be slow.

Note I said 'inheritable'; one of the main flaws w/ version 0 is that it requires quite a few entries per CPV, instead of setting a default in the preamble and then overriding as needed at the CPV level.


What I'm suggesting is a COMPRESSOR in the preamble, and individual cpv's override it if they're not that compressor.
As for zacs tool to try and generate new views of a repository via hardlinking/recreating the tree... frankly it's a bit of a hack. *Via DEFAULT_URI and relying on the hash, you can make a stable repository that is able to be updated in place without corrupting ongoing downloads- simply put, new additions to the repo don't perturb current DL's since the md5 is the same (hash collision chance is low enough that I don't care about it here).



> 3) PACKAGE_EXTENSION is overly verbose and unclear it's specifying

> the compressor too; it's intention is for compression, state it as

> such (I mention this in light of URI's existance where

> PACKAGE_EXTENSION would only be a hint of compressor)

>

> Re: #1, there is a decent set of optimizations I'm kicking around in

> pkgcore for the next version- a discussion should probably be started

> there.

>

> Offhand, having a compression specific header (a simple enumeration

> of known compressors) and a DEFAULT_URI that is python string



No go bro. The 'Packages' format should be independent of python.



> interpolation *assembled (for example,

> DEFAULT_URI="%(host)s/%(category)s/%(pf)s.txz") seems wiser. *Via

> doing what I'm suggesting, it would be possible to do binpkg

> repository 'views' w/out having to map each binpkg into the url space

> for it.

Then come up w/ an alternative w/ the same power as DEFAULT_URI that isn't python specific; think through the potentials of it, I could very easily centralize the binpkgs for an arch, use the hash as they're lookup value, then use the Packages cache as a 'view' into that binpkg repository. *Differing use flag combinations, differing license views, hell, differing ACCEPT_KEYWORDS, all of that can have the raw pkgs stored centrally while just providing differing views into it- DEFAULT_URI lays the groundwork for it.
 
Old 06-01-2010, 09:37 PM
Zac Medico
 
Default Package compression header for binhosts

On 06/01/2010 02:22 PM, Brian Harring wrote:
> As for zacs tool to try and generate new views of a repository via
> hardlinking/recreating the tree... frankly it's a bit of a hack. Via
> DEFAULT_URI and relying on the hash, you can make a stable repository that
> is able to be updated in place without corrupting ongoing downloads- simply
> put, new additions to the repo don't perturb current DL's since the md5 is
> the same (hash collision chance is low enough that I don't care about it
> here).

When you say "hash collision" are you talking about
http://crosbug.com/3225? Maybe that behavior is acceptable for
small-scale private use, but for large scale public repositories I'd
say it's totally unacceptable. Eventually, I'd like to see gentoo
officially distributing binary packages, so that we'll be able to
get a slice of the binary distribution pie. When that happens, we're
certainly not going to want to have race conditions like these in
our public binhosts.
--
Thanks,
Zac
 
Old 06-01-2010, 09:52 PM
Brian Harring
 
Default Package compression header for binhosts

On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico <zmedico@gentoo.org> wrote:

On 06/01/2010 02:22 PM, Brian Harring wrote:

> As for zacs tool to try and generate new views of a repository via

> hardlinking/recreating the tree... frankly it's a bit of a hack. *Via

> DEFAULT_URI and relying on the hash, you can make a stable repository that

> is able to be updated in place without corrupting ongoing downloads- simply

> put, new additions to the repo don't perturb current DL's since the md5 is

> the same (hash collision chance is low enough that I don't care about it

> here).



When you say "hash collision" are you talking about

http://crosbug.com/3225? Maybe that behavior is acceptable for

small-scale private use, but for large scale public repositories I'd

say it's totally unacceptable.
That bug isn't about a collision, it's about files being replaced underneath Packages feet. *Even with the tricks you've leveled the issue of things changing under foot still is possible- you've just made the race less likely.

What I was talking about was solving this issue once and for all via restructuring, and specifically refering to the potential of an md5 collision in the URI space- specifically what I'm implementing for pkgcore is the ability to do stupid stuff like this-

http://host/binpkg-store/$MD5.{txz,tbz2,tgz}
then have multiple views accessible just via pointing the binpkg repo remote url at

http://host/views/license/oss-approved/http://host/views/keywords/amd64/stable/
http://host/views/raw/ # no filtering on the view of the binpkg repo, see everything.
Via restructuring where the binpkgs are stored and doing this approach, multiple views can be had easily into the repo. *An additional benefit of this approach is that via making URI able to point outside the host, you could combine multiple seperate repositories into one just via a view.

* Eventually, I'd like to see gentoo

officially distributing binary packages, so that we'll be able to

get a slice of the binary distribution pie. When that happens, we're

certainly not going to want to have race conditions like these in

our public binhosts.

I'd suggest abandoning the current repository layout of Packages then, since it's irrevocably flawed. *You can hack around it via jamming timestamp/md5 info into URI, but that's not a sane solution.

~harring
 
Old 06-01-2010, 11:53 PM
Zac Medico
 
Default Package compression header for binhosts

On 06/01/2010 02:52 PM, Brian Harring wrote:
> That bug isn't about a collision, it's about files being replaced underneath
> Packages feet. Even with the tricks you've leveled the issue of things
> changing under foot still is possible- you've just made the race less
> likely.

AFAIK the race is completely eliminated by the RCU-like snapshot
mechanism. I think I like your hash-in-the-filename idea better
though, since it seems simpler to implement and maintain. It can
even be done with the existing version 0 format by abusing the
per-package PATH attribute to refer to a filename that contains a
hash (maybe something like $CATEGORY/$PF.$MD5.tbz2). I wouldn't want
to abuse PATH for compression in version 0 though, since clients are
only required to support tbz2.

> What I was talking about was solving this issue once and for all via
> restructuring, and specifically refering to the potential of an md5
> collision in the URI space- specifically what I'm implementing for pkgcore
> is the ability to do stupid stuff like this-
>
> http://host/binpkg-store/$MD5.{txz,tbz2,tgz}

That would be the MD5 of the entire file, after compression and
having the xpak segment appended, right?

> then have multiple views accessible just via pointing the binpkg repo remote
> url at
>
> http://host/views/license/oss-approved/
> http://host/views/keywords/amd64/stable/
> http://host/views/raw/ # no filtering on the view of the binpkg repo, see
> everything.

So, the default path of a package would come from looking at the MD5
in the Packages file and then mapping that to a path?

> Via restructuring where the binpkgs are stored and doing this approach,
> multiple views can be had easily into the repo. An additional benefit of
> this approach is that via making URI able to point outside the host, you
> could combine multiple seperate repositories into one just via a view.

This might also be useful for creating per-profile views while
allowing packages to be shared between profiles in cases when
hosting a separate build would be redundant. It might be possible to
save lots of build time, disk space, and testing that way.

Being able to have multiple builds of the same package with
different USE settings is also solves bug 150031 [1].

>> Eventually, I'd like to see gentoo
>> officially distributing binary packages, so that we'll be able to
>> get a slice of the binary distribution pie. When that happens, we're
>> certainly not going to want to have race conditions like these in
>> our public binhosts.
>>
>
> I'd suggest abandoning the current repository layout of Packages then, since
> it's irrevocably flawed. You can hack around it via jamming timestamp/md5
> info into URI, but that's not a sane solution.

Shrug, it's a handy way to solve race conditions given the existing
version 0 format. It's not optimal, so we'll surely want something
better in version 1.

[1] http://bugs.gentoo.org/show_bug.cgi?id=150031
--
Thanks,
Zac
 
Old 06-02-2010, 03:42 AM
Brian Harring
 
Default Package compression header for binhosts

On Tue, Jun 01, 2010 at 04:53:31PM -0700, Zac Medico wrote:
> On 06/01/2010 02:52 PM, Brian Harring wrote:
> > That bug isn't about a collision, it's about files being replaced underneath
> > Packages feet. Even with the tricks you've leveled the issue of things
> > changing under foot still is possible- you've just made the race less
> > likely.
>
> AFAIK the race is completely eliminated by the RCU-like snapshot
> mechanism. I think I like your hash-in-the-filename idea better
> though, since it seems simpler to implement and maintain.

You're forgetting about how one actually updates the snapshot- client
grabs the packages cache, starts pulling binpkgs. During that time
the snapshot is being updated- the client now has a stale view of the
repo, and since the repo's structure is based on cpv (which doesn't
change regardless of metadata changing like use configuration) they
can grab a binpkg that has the wrong metadata/checksums.

It's racey in exactly the same was as before, the only difference is
you switched it to rewrite the tbz2 in a temp file instead of directly
to the tbz2. Reduction, but same level of risk for any form of
updates.

Snapshot script just duck tapes around the issue, while leaving the
core flaw intact.

> > What I was talking about was solving this issue once and for all via
> > restructuring, and specifically refering to the potential of an md5
> > collision in the URI space- specifically what I'm implementing for pkgcore
> > is the ability to do stupid stuff like this-
> >
> > http://host/binpkg-store/$MD5.{txz,tbz2,tgz}
>
> That would be the MD5 of the entire file, after compression and
> having the xpak segment appended, right?

Yep. The only potential issue here is the unlikely case of a CHF
collision. There is a way to resolve that one too, although it is
outside of what I'm willing to do format wise (namely a secondary url
fallback).



> > then have multiple views accessible just via pointing the binpkg repo remote
> > url at
> >
> > http://host/views/license/oss-approved/
> > http://host/views/keywords/amd64/stable/
> > http://host/views/raw/ # no filtering on the view of the binpkg repo, see
> > everything.
>
> So, the default path of a package would come from looking at the MD5
> in the Packages file and then mapping that to a path?

default path would be defined in the preamble by a string interpolated
pattern; whatever folk wanted to use.

A sane default is %(host)s/raw-pkgs/%(md5)s.%(compressor_ext)s imo.


> > Via restructuring where the binpkgs are stored and doing this approach,
> > multiple views can be had easily into the repo. An additional benefit of
> > this approach is that via making URI able to point outside the host, you
> > could combine multiple seperate repositories into one just via a view.
>
> This might also be useful for creating per-profile views while
> allowing packages to be shared between profiles in cases when
> hosting a separate build would be redundant. It might be possible to
> save lots of build time, disk space, and testing that way.
>
> Being able to have multiple builds of the same package with
> different USE settings is also solves bug 150031 [1].

Yep and Yep.

> >> Eventually, I'd like to see gentoo
> >> officially distributing binary packages, so that we'll be able to
> >> get a slice of the binary distribution pie. When that happens, we're
> >> certainly not going to want to have race conditions like these in
> >> our public binhosts.
> >>
> >
> > I'd suggest abandoning the current repository layout of Packages then, since
> > it's irrevocably flawed. You can hack around it via jamming timestamp/md5
> > info into URI, but that's not a sane solution.
>
> Shrug, it's a handy way to solve race conditions given the existing
> version 0 format. It's not optimal, so we'll surely want something
> better in version 1.

The problem here is that version0 still maps down to the existing
binpkg on disk layout. That layout is the core flaw here- as long as
binpkgs are stored cpv orientated, version0 isn't able to do the crazy
things I'm intending.
~harring
 

Thread Tools




All times are GMT. The time now is 06:12 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org