FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 01-28-2010, 02:17 PM
Beber
 
Default Building custom package for multi-arch/system

Hi,

This mail is continuation of a discussion last month on IRC on
#gentoo-portage. As I think it's larger than just portage array, here it
is.

I have one server dedicated for compilation (xm-compile), and many
applicative servers (xmred5, lemonhead, etc).

Currently I use the --buildpkg on xm-compile to build packages
emergeable by --getbinpkgonly. Everything is fine except that I'm not
able to --getbinpkg function of USE flags, CFLAGS, CHOST nor ACCEPT_KEYWORDS.

So all packages on all servers are build in the same way with same USE.
But I would like to avoid it. I have mx servers (exim) that need mysql
lib, so mysql USE is there. I have mysql servers so -minimal USE is
there. I don't want to install a full mysql on mx servers, I just need
mysql lib.

So I'd like to build custom packages for almost everything.

I don't want to build packages on servers for security reasons (somes
could reproach me that, but it's not the topic and for a limitation
of disk usage.

So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
names function of USE, like :
CHOST="x86_64-pc-linux-gnu"
ACCEPT_KEYWORDS="amd64"
tcpdump

USE="+chroot +ipv6 -ssl -test -samba -smi"
is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7 b4416738.tbz2

USE="+chroot +ipv6 +ssl -test +samba -smi"
is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb 495980bd.tbz2

Like this, I can easily add on client side a portage bashrc to get the file
and place it in the good place to don't have portage to download it as
it would find the right file with good USE. This implies I need to
emerge 1 package per 1 package which is horrible after some times.

So, do you guys plan to implement a such thing ? That's one of the
features that is mostly missing imho. The principal miss in on client
side as I have tools to manage packages but would like to not have too
much specific scripts on client side.

Thanks,

--
Beber
 
Old 01-28-2010, 04:48 PM
Jacob Godserv
 
Default Building custom package for multi-arch/system

(I'm not an official developer.)

On Thu, Jan 28, 2010 at 10:17, Beber <beber@meleeweb.net> wrote:
> So, do you guys plan to implement a such thing ? That's one of the
> features that is mostly missing imho. The principal miss in on client
> side as I have tools to manage packages but would like to not have too
> much specific scripts on client side.

This is also high on my "things to hack" list. (Not that I'd ever come
up with something acceptable, but it's worth a try.

I would love to see the whole PKGDIR variable vanish altogether and
merge binary builds into ebuilds, creating a sort of apt-get / portage
mixture, all with a "standard" ebuild setup. By allowing binaries to
take on all the functionality of ebuilds, features like USE flags,
multiple versions per repository, varying CFLAGs, and other such
features are a given. The binary can be downloaded via standard ebuild
methods, even.

Something to chew on.

--

"For then there will be great distress, unequaled
from the beginning of the world until now — and never
to be equaled again. If those days had not been cut
short, no one would survive, but for the sake of the
elect those days will be shortened."

Are you ready?
 
Old 01-28-2010, 07:07 PM
Gilles Dartiguelongue
 
Default Building custom package for multi-arch/system

Hi,

I have a somewhat similar need and to satisfy it, I'm building packages
in a chroot that has all the common packages then one chroot per
variation. It's probably nowhere scalable enough for you though and
would most likely but much more efficient if package building could be
wrapped so that it builds the package for one setup, then reconfigures
the package and build it for the new setup without cleaning everything
in the middle. Some scripting could probably do that but then I think
this joins well with multiple ABIs discussions as it basically means
portage needs to be able to build the same package several times with
different configuration.

--
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo
 
Old 01-28-2010, 09:31 PM
"Robin H. Johnson"
 
Default Building custom package for multi-arch/system

On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> names function of USE, like :
> CHOST="x86_64-pc-linux-gnu"
> ACCEPT_KEYWORDS="amd64"
> tcpdump
>
> USE="+chroot +ipv6 -ssl -test -samba -smi"
> is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7 b4416738.tbz2
>
> USE="+chroot +ipv6 +ssl -test +samba -smi"
> is placed under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb 495980bd.tbz2
A word of warning on this.

Simply capturing the USE flags is not enough, the libraries that a
binary is linked against can cause problems as well, as the dependency
imposed by linking is much stricter than the RDEPEND in the ebuild.

tcpdump for example has:
RDEPEND="... ssl? ( >=dev-libs/openssl-0.9.6m )"

But if you built and linked against openssl-0.9.7*, then upgraded to
openssl-0.9.8, you can expect breakage.

Similar things happened with binaries linked against libmysqlclient in
the past.

As to my own take on a solution to the problem for work stuff, we simply
maintain one chroot per unique environment (we only have 5 envs), and
build in the chroot, then use those binpkgs on the boxes.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
 
Old 01-28-2010, 10:23 PM
Philipp Riegger
 
Default Building custom package for multi-arch/system

On Thu, 28 Jan 2010 16:17:41 +0100
Beber <beber@meleeweb.net> wrote:

> So, I did a wrapper to emerge --buildpkg to create tbz2 with SHA1 in
> names function of USE, like :
> CHOST="x86_64-pc-linux-gnu"
> ACCEPT_KEYWORDS="amd64"
> tcpdump
>
> USE="+chroot +ipv6 -ssl -test -samba -smi"
> is placed
> under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+2bf4bfffad82d4ae519f76770b4f7db7 b4416738.tbz2
>
> USE="+chroot +ipv6 +ssl -test +samba -smi"
> is placed
> under /data/pkg/amd64/x86_64-pc-linux-gnu/net-analyzer/tcpdump-4.0.1_pre20090709+5e099e8337e4cadfceda6ccf4c881fbb 495980bd.tbz2
>

So these are the SHA1 sums of the packages? Nice.

I would like to see something like this:

With different CFLAGS, CHOST etc. the user should set a seperate PKGDIR.

USE-flags are encoded in the filename:

Sort USE-flags alphabetically, use a 1 for every enabled and a 0 for
every disabled USE-flag, make this array smaller using hex code ore
base64.

> USE="+chroot +ipv6 -ssl -test -samba -smi"

110000

tcpdump-4.0.1_pre20090709+c0.tbz2

> USE="+chroot +ipv6 +ssl -test +samba -smi"

111010

tcpdump-4.0.1_pre20090709+e2.tbz2

Problem with this is, if anything with the USEflags as we see it from
outside changes (USE-flags are added, deletes, renamed and therefore
change position), this gets none-unique, which can be fixed in several
ways.

Philipp
 
Old 01-29-2010, 04:24 AM
Max Arnold
 
Default Building custom package for multi-arch/system

On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> So, do you guys plan to implement a such thing ? That's one of the
> features that is mostly missing imho. The principal miss in on client
> side as I have tools to manage packages but would like to not have too
> much specific scripts on client side.

I like the way it done in OpenEmbedded. You have the tree of recipes (think of portage tree)
and bunch of targets. For each target BitBake can generate binary release and package feed.
Client package management is lightweight and does not require BitBake, recipes tree and even
python. At least this is my lame interpretation of how it works

Maybe this "metadistribution" approach is cleaner than binary package support in emerge. If
user wants to compile packages on the client, he uses portage. If not - he can setup build
server for multiple targets and completely drop portage from client machines. The only thing
client should know is feed url with full list of binary packages. And I do not think client
should deal with USE flags - for large installations unification is the only sane way to scale.
 
Old 01-29-2010, 08:28 AM
Antoni Grzymala
 
Default Building custom package for multi-arch/system

Max Arnold dixit (2010-01-29, 12:24):

> On Thu, Jan 28, 2010 at 04:17:41PM +0100, Beber wrote:
> > So, do you guys plan to implement a such thing ? That's one of the
> > features that is mostly missing imho. The principal miss in on
> > client side as I have tools to manage packages but would like to not
> > have too much specific scripts on client side.
>
> I like the way it done in OpenEmbedded. You have the tree of recipes
> (think of portage tree) and bunch of targets. For each target BitBake
> can generate binary release and package feed. Client package
> management is lightweight and does not require BitBake, recipes tree
> and even python. At least this is my lame interpretation of how it
> works
>
> Maybe this "metadistribution" approach is cleaner than binary package
> support in emerge. If user wants to compile packages on the client, he
> uses portage. If not - he can setup build server for multiple targets
> and completely drop portage from client machines. The only thing
> client should know is feed url with full list of binary packages. And
> I do not think client should deal with USE flags - for large
> installations unification is the only sane way to scale.

I think something similar is very nicely done on R-Path based
distributions (with "flavours" [1]). Conary itself also seems to be a
pretty well thought out package manager.

[1] http://docs.rpath.com/conary/Conaryopedia/sect-flavors.html

--
[a]
 
Old 01-29-2010, 09:52 AM
Philipp Riegger
 
Default Building custom package for multi-arch/system

On Thu, 28 Jan 2010 22:31:04 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> But if you built and linked against openssl-0.9.7*, then upgraded to
> openssl-0.9.8, you can expect breakage.

But can't this be kind of ignored and solves the Gentoo way? If the
binpkg does not work, don't install it and simply rebuild it (if not
stated otherwise on the command line)?

Philipp
 
Old 01-29-2010, 08:44 PM
"Robin H. Johnson"
 
Default Building custom package for multi-arch/system

On Fri, Jan 29, 2010 at 11:52:40AM +0100, Philipp Riegger wrote:
> On Thu, 28 Jan 2010 22:31:04 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> > But if you built and linked against openssl-0.9.7*, then upgraded to
> > openssl-0.9.8, you can expect breakage.
> But can't this be kind of ignored and solves the Gentoo way? If the
> binpkg does not work, don't install it and simply rebuild it (if not
> stated otherwise on the command line)?
The gentoo "way" in this case is revdep-rebuild, and you've already
stated that your servers do not have a compiler.

You need to either expose the linkage dependencies outside the binpkg,
or introspect it when you're about to install it to find that it's NOT
going to work.

zmedico/portage folk: Can we have a phase during binpkg install, that
lets us check ELF linkages before the files get moving to ${ROOT}?

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
 
Old 01-29-2010, 09:20 PM
Zac Medico
 
Default Building custom package for multi-arch/system

On 01/29/2010 01:44 PM, Robin H. Johnson wrote:
> zmedico/portage folk: Can we have a phase during binpkg install, that
> lets us check ELF linkages before the files get moving to ${ROOT}?

Yeah, portage-2.2 has a LinkageMap class that we can use to verify
that all the needed sonames are present.
--
Thanks,
Zac
 

Thread Tools




All times are GMT. The time now is 03:27 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org