FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 02-16-2009, 02:50 PM
Kari Pahula
 
Default Proposed release goal: fix debian/rules build-arch

Currently, Debian Policy doesn't match with the current practice in
section 7.7.

> The Build-Depends-Indep and Build-Conflicts-Indep fields must be
> satisfied when any of the following targets is invoked: build,
> build-indep, binary and binary-indep.

I know that people like to say that Policy should reflect reality, not
have wishful thinking (like in #178809). It's been tried before but
I'd like to try again to get a transition done to make reality into
what 7.7 says it is.

As it stands, buildds call "debian/rules build" without having B-D-I
installed, contrary to 7.7. Buildds do call "debian/rules
binary-arch". As such, B-D-I does what it's supposed to do on buildds
only in conjunction with binary-arch and clean targets.

Obviously, we'd get half of the archive FTBFS if we made buildds call
build-arch instead of build now, since it's an optional feature.
Having a call to "debian/rules build-arch" fail tells us less than
we'd like, too. Nobody's yet written a reliable check to see if the
build-arch target is present in a debian/rules file. I'm hoping that
we're only dealing with debian/rules files that are makefiles. It'd
be desirable to know if build-arch is supported from the .dsc file
alone, without unpacking the source package.

This has been discussed before and I'll list some options from the top
of my head. I'm hoping to start a discussion, again.

Now, option 1 (cold turkey):

Make build-arch to be a mandatory target in debian/rules files in the
next policy version (3.9.0, already?). Any existing "build" targets
work, by necessity, correctly without having B-D-I installed, and a
debian/rules file could be fixed by adding one line:
build-arch: build

Buildds would still call "debian/rules build" on anything that had a
smaller Standards-Version than 3.9.0, and "debian/rules build-arch" on
the rest. It'd be the maintainer's responsibility to check that it
works before bumping the version.

Option 2 (features field):

Add a field called "Build-Features:" to debian/control and have it
contain a space separated list of tokens. Define "build-arch" as a
recognized value. Put that in .dsc. As with things like this, we'd
potentially get stuck with it forever, but it'd be the least invasive
thing to do and still get buildds to use build-arch. There'd be no
transition, other than getting sbuild patched.

We could also change build-arch into a "should" feature and warn
anyone who didn't support build-arch and switch over to having it as a
"must" once everyone did it. It'd be easy to check for that, with
this.

Option 3 (lintian hackery first):

I know some people would like to see a lintian check, first. The
thing is, debian/rules is a program, so trying to figure out any
properties about it, like "does it support feature X?" or "does it
halt?" gets quite close to the halting problem.

GNU make does have some options to query a makefile, like --dry-run
and --question. Even those would require evaluating a makefile, at
least to some degree. If someone put $(shell rm *) in there, it'd do
that. I'd like to see something like "Running debian/rules --dry-run
or --question must not have harmful side effects or affect any
subsequent calls to debian/rules." in the policy before relying on
those. I'm hoping that there's nothing controversial about this
addition. IMHO it's a rather pathological makefile that does things
like that.

I'd like to see some explicit option added to make that would just
answer the question "Is there a rule that would match target X?".
Same considerations about evaluation would apply as to -n and -q. As
it stands, I'd need to do something like this:

debian/rules -q build-arch 2>&1 >/dev/null | tail -n 1 |
egrep -e '^make: *** No rule to make target `build-arch.. Stop.$'

As an aside, guaranteeing that --dry-run wouldn't do anything evil
would help lintian in other matters, too. For example, this is what
the current version does to check whether a package uses CDBS:
if (m,^includes+/usr/share/cdbs/1/rules/debhelper.mk,)

I'd rather run "debian/rules -nds" to check if it includes something
from under /usr/share/cdbs/.

Option 4 (give up):

Remove the mention of "build" from 7.7. Policy would match the
current usage, right then. This is not what I'd like to see, since I
think that a reliable build-arch would be a really nice thing to have.

Option 5 (further discussion):

Do nothing. Have this discussion again sometime after Squeeze. Can
we please not do this?


I see less of a need for a build-indep target. Should we touch that?
 
Old 02-16-2009, 04:15 PM
Raphael Hertzog
 
Default Proposed release goal: fix debian/rules build-arch

On Mon, 16 Feb 2009, Kari Pahula wrote:
> Currently, Debian Policy doesn't match with the current practice in
> section 7.7.
>
> > The Build-Depends-Indep and Build-Conflicts-Indep fields must be
> > satisfied when any of the following targets is invoked: build,
> > build-indep, binary and binary-indep.
>
> I know that people like to say that Policy should reflect reality, not
> have wishful thinking (like in #178809). It's been tried before but
> I'd like to try again to get a transition done to make reality into
> what 7.7 says it is.

I don't know if you are aware, but there's lots of discussion already
about this in various dpkg bug reports and in particular this one:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=229357

This bug is on my radar and I certainly plan to fix it in the squeeze
timeline but before we can clearly fix it, we need to come to a reasonable
agreement about what constitutes the official interface to build Debian
packages and how it should look like. We currently have an unfortunate
divergence between dpkg-buildpackage and the policy that needs to be
solved before we can tackle this problem seriously.

> Now, option 1 (cold turkey):
>
> Make build-arch to be a mandatory target in debian/rules files in the
> next policy version (3.9.0, already?). Any existing "build" targets
> work, by necessity, correctly without having B-D-I installed, and a
> debian/rules file could be fixed by adding one line:
> build-arch: build
>
> Buildds would still call "debian/rules build" on anything that had a

Note: buildd use dpkg-buildpackage so the change needs to be done in
dpkg-buildpackage.

> Option 2 (features field):
>
> Add a field called "Build-Features:" to debian/control and have it
> contain a space separated list of tokens. Define "build-arch" as a
> recognized value. Put that in .dsc. As with things like this, we'd
> potentially get stuck with it forever, but it'd be the least invasive
> thing to do and still get buildds to use build-arch. There'd be no
> transition, other than getting sbuild patched.
>
> We could also change build-arch into a "should" feature and warn
> anyone who didn't support build-arch and switch over to having it as a
> "must" once everyone did it. It'd be easy to check for that, with
> this.

My current plan is to implement a Build-Options field but the expected use
case for such a field are a bit too broad and it leads us to the
discussion about how much "build customization" we should support and how
that customization must be handled (and how we should mix choices of
packagers, choices of user that rebuild the package, choices of the
(derivative) distributions). Note the usage of standards-version to
auto-enable some options is still possible even if we implement
Build-Options.

The first and main customization that users and derivatives distributions
want is related to compiler options so that they can do hardened builds or
optimized builds without having to hand-edit each and every package.
We have integrated some changes during Lenny related to that but it can't
deliver its potential if we don't change the policy to say that package
must respect/use the compiler flags provided in the environment.
Unfortunately, that change was merged/made in dpkg without much
consultation of debian-policy and we're now stuck in a situation where the
disagreement expressed afterwards would certainly lead to a refusal of a
policy change going in that direction.

On that topic I recently started to draft this wiki page, I hope it can
help summarize the options and the issues with each approach and we can
decide what direction we want to take.

http://wiki.debian.org/Teams/Dpkg/DebianRules

This wiki page is centered only on the topic of the preparation of the
build environment but its outcome will obviously have some consequences
on the decision we're going to take for the the build vs build-arch
problem. At least because it will have implications on the implementation
of Build-Options.

(I'm not sure that this discussion will be very constructive at this
stage because I wanted to get some more private feedback to better channel
the discussion but let's see…)

Cheers,
--
Raphaël Hertzog

Le best-seller français mis * jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2009, 06:21 PM
Russ Allbery
 
Default Proposed release goal: fix debian/rules build-arch

Kari Pahula <kaol@debian.org> writes:

> I know some people would like to see a lintian check, first. The thing
> is, debian/rules is a program, so trying to figure out any properties
> about it, like "does it support feature X?" or "does it halt?" gets
> quite close to the halting problem.
>
> GNU make does have some options to query a makefile, like --dry-run and
> --question. Even those would require evaluating a makefile, at least to
> some degree. If someone put $(shell rm *) in there, it'd do that. I'd
> like to see something like "Running debian/rules --dry-run or --question
> must not have harmful side effects or affect any subsequent calls to
> debian/rules." in the policy before relying on those. I'm hoping that
> there's nothing controversial about this addition. IMHO it's a rather
> pathological makefile that does things like that.

Such a requirement unfortunately still won't mean that Lintian can use
that option to do a check of debian/rules. As long as make is willing to
run such code, we can't just rely on a Policy statement saying that you're
not supposed to do that. It is, among other things, a security problem.
You should be able to run Lintian on a package of unknown origin and
provenance and not have it be able to do things that you don't want. What
we would need is a make option that ensures that make would never run such
code, and unfortunately that's likely to lead to false positives in some
of the stranger cases.

Lintian can go a long way by scanning debian/rules. This mostly fails
with build systems like CDBS (but one can generally assume that they'll do
the right thing) and hand-rolled build systems involving includes or more
fancy trickery. Lintian can't get the latter right, but they're also
fairly rare.

There are also the few packages in the archive that don't have a makefile
as debian/rules. I've been tempted for some time to file RC bugs against
all of them.

http://lintian.debian.org/tags/debian-rules-not-a-makefile.html

> Option 4 (give up):
>
> Remove the mention of "build" from 7.7.

I assume you mean build-arch and build-indep here. I would go a step
further and deprecate Build-{Depends,Conflicts}-Indep if we go that route,
remove the corresponding Lintian checks, and start letting people simplify
their debian/control files.

> Policy would match the current usage, right then. This is not what I'd
> like to see, since I think that a reliable build-arch would be a really
> nice thing to have.

I have to admit that I'm tempted by this approach, mostly because it's not
clear to me that the build-arch vs. build-indep separation actually gains
us anything that useful. The point, so far as I can tell, is to save
buildd time by not building the architecture-independent packages. How
much time would we actually be saving? Is it worth putting a lot of human
effort into making that situation possible? Generally CPU cycles are far,
far cheaper than human cycles.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2009, 07:16 PM
Kari Pahula
 
Default Proposed release goal: fix debian/rules build-arch

On Mon, Feb 16, 2009 at 11:21:42AM -0800, Russ Allbery wrote:
> Such a requirement unfortunately still won't mean that Lintian can use
> that option to do a check of debian/rules. As long as make is willing to
> run such code, we can't just rely on a Policy statement saying that you're
> not supposed to do that. It is, among other things, a security problem.

That's a good point, but not running debian/rules means that you'd be
making it a requirement to write debian/rules files in a stylised way,
to make it greppable. Conventions are one thing, that'd be another.
That'd have a human cost too. But this is somewhat coincidental to
this. Coming up with a test, even an imperfect one, could help push
changes forwards.

> I have to admit that I'm tempted by this approach, mostly because it's not
> clear to me that the build-arch vs. build-indep separation actually gains
> us anything that useful. The point, so far as I can tell, is to save
> buildd time by not building the architecture-independent packages. How
> much time would we actually be saving? Is it worth putting a lot of human

Ask buildd admins. They could start downloading and installing B-D-I
along with B-D today. Deprecating B-D-I and -arch and -indep would be
a small step after that.

> effort into making that situation possible? Generally CPU cycles are far,
> far cheaper than human cycles.

Another thing that B-D-I is good for: breaking dependency cycles. An
example from the upcoming version of ghc6: ghc6 uses haddock to build
API docs. Haddock needs to be built with the same version of ghc6 it
generates docs for. Putting haddock in ghc6's B-D-I avoids that
cycle.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2009, 10:34 PM
Steve Langasek
 
Default Proposed release goal: fix debian/rules build-arch

On Mon, Feb 16, 2009 at 11:21:42AM -0800, Russ Allbery wrote:
> There are also the few packages in the archive that don't have a makefile
> as debian/rules. I've been tempted for some time to file RC bugs against
> all of them.

> http://lintian.debian.org/tags/debian-rules-not-a-makefile.html

Interestingly, all but one of these is a false positive, at least in the
sense of whether debian/rules is a makefile. The vdr packages don't use
/usr/bin/make as the interpreter line, but debian/rules *is* a makefile -
they just have a rather convoluted custom script that they use to set up the
environment before calling make.

That leaves just 'leave' which is using a shell script as debian/rules.
Every time this issue has come up before, Josip has stuck to his guns on
using a non-makefile for this package; but it is a policy violation, and if
being able to rely on debian/rules being a makefile helps us finally unblock
the build-arch mess, I don't think it's defensible. I'm all in favor of
enforcing this policy dictum as RC for squeeze.

> > Policy would match the current usage, right then. This is not what I'd
> > like to see, since I think that a reliable build-arch would be a really
> > nice thing to have.

> I have to admit that I'm tempted by this approach, mostly because it's not
> clear to me that the build-arch vs. build-indep separation actually gains
> us anything that useful. The point, so far as I can tell, is to save
> buildd time by not building the architecture-independent packages. How
> much time would we actually be saving? Is it worth putting a lot of human
> effort into making that situation possible? Generally CPU cycles are far,
> far cheaper than human cycles.

In some cases, building the arch-indep documentation takes longer, and
requires downloading/installing more build dependencies, than building the
arch-dep binaries. I've found this to be a waste of human cycles before
when building packages locally: since it's not possible to bypass the
"build-indep" component in a sane fashion, I wind up waiting on the
arch-indep bits when trying to test out a patch that only affects the
arch-dependent parts of the package. If the doc building is expensive
enough to be noticeable to me when building on amd64, I would imagine that
the impact on buildds (and hand builds) for slower archs is significant,
too.

If we can ever settle on a suitable implementation, I would expect the
savings of both human and CPU cycles to be sizeable, and worth the effort.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2009, 11:05 PM
Russ Allbery
 
Default Proposed release goal: fix debian/rules build-arch

Steve Langasek <vorlon@debian.org> writes:

> Interestingly, all but one of these is a false positive, at least in the
> sense of whether debian/rules is a makefile. The vdr packages don't use
> /usr/bin/make as the interpreter line, but debian/rules *is* a makefile
> - they just have a rather convoluted custom script that they use to set
> up the environment before calling make.

Which means that if you call make on it directly, it doesn't work, since
the environment variables aren't set and all the stuff that shell script
does then isn't done. I don't really agree that it's a false positive,
although I agree that it will happen to work given how we currently build
packages.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-16-2009, 11:28 PM
Charles Plessy
 
Default Proposed release goal: fix debian/rules build-arch

Le Mon, Feb 16, 2009 at 10:16:56PM +0200, Kari Pahula a écrit :
>
> Another thing that B-D-I is good for: breaking dependency cycles. An
> example from the upcoming version of ghc6: ghc6 uses haddock to build
> API docs. Haddock needs to be built with the same version of ghc6 it
> generates docs for. Putting haddock in ghc6's B-D-I avoids that
> cycle.

Hello everybody,

maybe Build-Recommends could also solve this…

Have a nice day,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-17-2009, 05:51 PM
Cyril Brulebois
 
Default Proposed release goal: fix debian/rules build-arch

Charles Plessy <plessy@debian.org> (17/02/2009):
> maybe Build-Recommends could also solve this…

Does “build reproducibility” mean something to you? I seem to have read
about it several times in the previous days already. And that wasn't the
first time it's been mentioned.

Mraw,
KiBi.
 
Old 02-18-2009, 12:20 AM
Charles Plessy
 
Default Proposed release goal: fix debian/rules build-arch

Le Tue, Feb 17, 2009 at 07:51:14PM +0100, Cyril Brulebois a écrit :
>
> Does “build reproducibility” mean something to you?

Hi Cyril,

Build reproduciblitity means to me that two instances package built in the same
environment should be reasonably identical (things like timestamps or random
numbers will never be reproducible).

Interestingly, the Debian packages are built in an environment where
reproducibility is only transient as Sid is constantly updated. This is even a
feature in the case of binNMUs.

I do not think that anybody proposes a Build-Recommend field that can result in
binary differences in the context of the Debian buildd network. However, since
I agree with the persons who question the usefulness of distinguishing
Build-Indep dependancies, because dropping this distinction would make my work
easier and therefore more robust, I try to figure out a possible alternative.

For the purpose of skipping documentation building and test making, be it
locally or remotely, Build-Recommends can be a useful alternative to
Build-Depends-Indep. In particular, the use of Build-Depends-Indep to emulate a
"nodoc" option is only possible if the documentation is separated from the main
package in an Arch:all package. For some packages, for instance altree, it
would mean to make a -doc package with a single PDF file in, for the sake of
removing TeX and LaTeX from the build-dependancies.

I hope that this also anwers to Steve.

Have a nice day,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 02-18-2009, 03:11 AM
Paul Wise
 
Default Proposed release goal: fix debian/rules build-arch

On Wed, Feb 18, 2009 at 3:51 AM, Cyril Brulebois <kibi@debian.org> wrote:
> Charles Plessy <plessy@debian.org> (17/02/2009):
>> maybe Build-Recommends could also solve this…
>
> Does "build reproducibility" mean something to you?

The combination of the simultaneous need for Build-Recommends and
build reproducibility will probably lead us to reimplementing
Gentoo-like source packages. IMO we should completely rethink the
"Debian source package" instead of cobbling stuff like
Build-Recommends on top of it.

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 09:24 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org