This is a proposal to introduce a common set of compiler options which
can be set independently from the package, and passed/injected to the
package build process. It was first discussed at the last UDS; a
corresponding wiki page can be found at [1].

== Rationale ==

The set of compiler/linker flags used for building packages is defined
in Debian policy:

* Packages should be compiled using '-g -O2' unless
DEB_BUILD_OPTIONS doesn't specify 'noopt' in which case packages
should be built using '-g -O0'.

* Packages should be stripped, unless 'nostrip' is passed in
DEB_BUILD_OPTIONS.

Currently two approaches are used to build the archive with modified
options:

* Changing the compiler specs to modify the default options (either
by patching the compiler or by adding custom spec files).

* Using scripts which wrap the invocations of the compiler and the
linker.

Changing the behaviour at this level has a few disadvantages:

* The default behaviour of the development tools is changed
unconditinally for all uses of the tools (not just package
building. -fstack-protector), or for the most use cases (i.e. let
build-essential depend on a package providing wrapper scripts or
spec files).

* Packages can hardly work around problems in the development tools
(for example lowering the optimization level, overriding a default
optimization level).

* It is errorprone to combine more than one wrapper package.

* The injected/changed options do not show up in the build log (in
the case of spec files), or require extra output (in the case of
wrapper scripts).

The injection of compiler/linker options should be done in the basic
package build tool (dpkg-buildpackage); the debian/rules file should
handle passing/overriding the options to the upstream build system.

Similiar implementations do exist for rpm (Fedora, OpenSuse) and the
Gentoo build system.

== Use Cases ==

* Optimize a distribution to build for a spcific processor (as
currently done with pentium-builder). Involves setting CFLAGS

* Build a distribution with a common set of security/hardening
options. Involves setting CPPFLAGS, CFLAGS, LDFLAGS

* Build shared libraries with -Bsymbolic-functions (no effect on
executables). Involves setting LDFLAGS.

* Test build a set of packages using a set of non-standard set of
compiler/linker options, overwriting any package specific
optimization options.

== Design ==

We define a set of macros which are passed from dpkg-buildpackage to
the debian build system (debian/rules). The injected flags are
collected and passed to the upstream build system (LoicMinier: is this
only done via the environment or via command line args as well?):

* 'DEB_HOST_CFLAGS': Optimization options which are passed to the
debian build system and can/should be overriden by the package
build if needed (default value: '-g -O2', or '-g -O0' if
'noopt' is specified. Overriding options can be used to explicitely
set a higher optimization level, or work around compiler bugs, which
only can be seen with some optimization levels (the last opt level "wins").

* 'DEB_HOST_CPPFLAGS': Preprocessor flags which are passed to the
debian build system and can/should be overriden by the package
build if needed (default: empty). This macro is seldom used (most
build systems just use 'CFLAGS' instead of 'CPPFLAGS').

* 'DEB_HOST_CXXFLAGS': Same as 'DEB_HOST_CFLAGS' for C++ sources.

* 'DEB_HOST_FFLAGS': Same as 'DEB_HOST_FFLAGS' for Fortran sources.

* 'DEB_HOST_LDFLAGS': Options passed to the compiler when linking
executables or shared objects (if the linker is called directly,
then '-Wl' and ',' have to be stripped from these
options. Default: empty.

* Compiler flags for other languages not mentioned above (Extend the
specification as needed). LoicMinier: would it make sense to have
libtool flags (LT_)? And configure args? What about setting the CC?

A second set of macros is used to overwrite package options, mostly
used for test rebuilds with changed cpu and optimization options.

* 'DEB_HOST_CFLAGS_APPEND': Optimization options appended to the
compiler flags, which must not be overwritten by the package
(mostly used to for test builds). Default value: empty.

* 'DEB_HOST_CPPFLAGS_APPEND': Preprocessor flags appended to the
preprocessor flags, which must not be overwritten by the package
(mostly used to for test builds). Default value: empty.

* 'DEB_HOST_CXXFLAGS_APPEND': Same as
'DEB_HOST_CFLAGS_APPEND' for C++ sources.

* 'DEB_HOST_FFLAGS_APPEND': Same as 'DEB_HOST_FFLAGS_APPEND'
for Fortran sources.

Note: The 'DEB_HOST_' prefix is used to define these options for
cross builds as well. For the case of a native build DEB_HOST ==
DEB_BUILD. We don't care about DEB_BUILD options for cross builds.

dpkg-buildpackage sets these flags to the default values unless the
flags are already defined in the environment.

=== Alternate naming of the macros ===

Do omit the 'DEB_HOST_' prefix and all packages should honour the
value of CFLAGS etc. in the environment. Many packages are likely to
do this by accident anyway. For packages that set CFLAGS etc., there
are basically two common cases:

* replace CFLAGS with -O0 etc. for debugging purposes

* add package-specific CFLAGS

In the first case, we could recommend a standard `debian/rules`
snippet to do that; `CFLAGS += -O0 -g` should be sufficient. In the
second case, `CFLAGS += WHATEVER` should work just fine.

This has the massive advantages that it does not require implementing
an extra variable for everything that might conceivably be used by
`make`, that it is already supported by the majority of build systems
in the archive provided that it is not explicitly overridden by
`debian/rules`, and that it is supported automatically by packages
with trivial build systems that use a configure which understands
CFLAGS, LDFLAGS, etc. A few exceptions will of course exist, but they
would exist with `DEB_HOST_CFLAGS` anyway. Besides, a large number of
the exceptions could be fixed simply by changing `=` to `+=`. To me,
this scheme seems a lot simpler and easier to deploy.

It has the disadvantage that these macros might be injected into the
build for some non-standard build systems.

== Implementation ==

The implementation itself is trivial, but needs to be done in every
source package.

TODO: Examples for debian/rules

TODO: Identify other build systems not using dpkg-buildpackage and
adopt these as well (autopkgtst).

TODO: Make build systems like cdbs comply with this spec.

== Test/Demo Plan ==

The implementation status of a package can be checked by
(automatically) inspecting the debian/rules file for every package
which builds architecture dependent packages, or by inspecting the
build dependencies of a source package (in the case that a build
system supporting the spec is used).

The correctness of the implementation can be checked by inspecting the
build logs.

== Outstanding Issues ==

?

---------------------------------------------------------------------

[1] https://wiki.ubuntu.com/DistCompilerFlags


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, Dec 04, 2007 at 05:08:13PM +0100, Matthias Klose wrote:
> We define a set of macros which are passed from dpkg-buildpackage to

If you say "macros" you mean the same as environment variables?

[...]
> === Alternate naming of the macros ===
>
> Do omit the 'DEB_HOST_' prefix and all packages should honour the
> value of CFLAGS etc. in the environment. Many packages are likely to
> do this by accident anyway. For packages that set CFLAGS etc., there
> are basically two common cases:
[...]
> It has the disadvantage that these macros might be injected into the
> build for some non-standard build systems.

If we would choose to use the common names without DEB_HOST_ prefix
should dpkg-buildpackage then honor them if they are already set in
the environment? Probably not as this seems error prone.

Gruesse,
--
Frank Lichtenheld <djpig@debian.org>
www: http://www.djpig.de/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Frank Lichtenheld writes:
> On Tue, Dec 04, 2007 at 05:08:13PM +0100, Matthias Klose wrote:
> > We define a set of macros which are passed from dpkg-buildpackage to
>
> If you say "macros" you mean the same as environment variables?

IIRC we cannot assume that debian/rules is a makefile and pass them as
macros directly, so we have to pass them as environment variables.

> [...]
> > === Alternate naming of the macros ===
> >
> > Do omit the 'DEB_HOST_' prefix and all packages should honour the
> > value of CFLAGS etc. in the environment. Many packages are likely to
> > do this by accident anyway. For packages that set CFLAGS etc., there
> > are basically two common cases:
> [...]
> > It has the disadvantage that these macros might be injected into the
> > build for some non-standard build systems.
>
> If we would choose to use the common names without DEB_HOST_ prefix
> should dpkg-buildpackage then honor them if they are already set in
> the environment? Probably not as this seems error prone.

yes, that makes sense.

Matthias


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wed, 5 Dec 2007 21:11:53 +0100, Matthias Klose <doko@cs.tu-berlin.de> said:

> Frank Lichtenheld writes:
>> On Tue, Dec 04, 2007 at 05:08:13PM +0100, Matthias Klose wrote:
>> > We define a set of macros which are passed from dpkg-buildpackage
>> > to
>>
>> If you say "macros" you mean the same as environment variables?

> IIRC we cannot assume that debian/rules is a makefile and pass them as
> macros directly, so we have to pass them as environment variables.

I think you remember incorrectly. Policy currently states
--8<---------------cut here---------------start------------->8---

4.9. Main building script: `debian/rules'
-----------------------------------------

This file must be an executable makefile, and contains the
package-specific recipes for compiling the package and building binary
package(s) from the source.
--8<---------------cut here---------------end--------------->8---



Also, note that other parts of policy also refer to the rules
being a makefile:
--8<---------------cut here---------------start------------->8---

4.6. Error trapping in makefiles
--------------------------------

When `make' invokes a command in a makefile (including your package's
upstream makefiles and `debian/rules'), it does so using `sh'.
....
--8<---------------cut here---------------end--------------->8---

So we can pass the macros directly on the command line, if you
so desire.

manoj
--
You don't have to know how the computer works, just how to work the
computer.
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Lars Wirzenius <liw@iki.fi> writes:

> The two packages in question are leave and webserver-package, and as it
> turns out, the latter is a false positive, since my grepping for
> debian/rules found a file that wasn't a real debian/rules file. So, only
> one package in etch/main uses something not a makefile for debian/rules.

http://lintian.debian.org/reports/Tdebian-rules-not-a-makefile.html

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Manoj Srivastava <srivasta <at> debian.org> writes:

>
> On Wed, 5 Dec 2007 21:11:53 +0100, Matthias Klose <doko <at> cs.tu-berlin.de>
said:
>
> > IIRC we cannot assume that debian/rules is a makefile and pass them as
> > macros directly, so we have to pass them as environment variables.
>
> I think you remember incorrectly. Policy currently states
[...]
> This file must be an executable makefile,

Indeed - but there's a bug, #432564, requesting to change it.

Regards, Frank


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

user srivasta@debian.org
usertag 432564 +dubious
thanks

On Thu, 6 Dec 2007 08:15:08 +0000 (UTC), Frank Küster <frank@debian.org> said:

> Manoj Srivastava <srivasta <at> debian.org> writes:
>>
>> On Wed, 5 Dec 2007 21:11:53 +0100, Matthias Klose <doko
>> <at> cs.tu-berlin.de>
> said:
>>
>> > IIRC we cannot assume that debian/rules is a makefile and pass them
>> > as macros directly, so we have to pass them as environment
>> > variables.
>>
>> I think you remember incorrectly. Policy currently states
> [...]
>> This file must be an executable makefile,

> Indeed - but there's a bug, #432564, requesting to change it.

That is the same as the older bug#88111, filed 1 Mar 2001. As
far as I can tell, both these bug are about a proposal that
a) Adds no practical value
b) does not represent current practice
c) not implementing the proposal is not a technical hindrance to any
package
d) stands in the way of technical proposals like passing information
to the build system on the command line
e) prevents people from relying on make semantics for builds.

The only reason for the bug report seems to be
a) because we can
b) aesthetics
c) profit???

I am convinced that this bug report is of dubious value, and if
Russ agrees, I am going to just mark this report as closed, or as
wontfix.

manoj
--
Is that really YOU that is reading this?
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Loïc Minier <lool@dooz.org> writes:
> On Thu, Dec 06, 2007, Manoj Srivastava wrote:

>> b) does not represent current practice
>> c) not implementing the proposal is not a technical hindrance to any
>> package

> This is the same point. Just for the record, there's a small set of
> packages not based on a Makefile for debian/rules.

To be precise, there's one package that doesn't use a makefile and a set
of around 14 that use a shell script wrapper around make. (I'm fairly
sure that what the latter set are doing could be done with an included
makefile, but I haven't checked to be sure.)

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>

On Thu, 6 Dec 2007 18:23:16 +0100, Loïc Minier <lool@dooz.org> said:

> On Thu, Dec 06, 2007, Manoj Srivastava wrote:
>> a) Adds no practical value

> It's about rejecting a change to policy; I don't see why it should
> add practical value.

The change was made in 2001. That is nearly 7 years ago. It
represents current practice. Get over it.

>> b) does not represent current practice
>> c) not implementing the proposal is not a technical hindrance to any
>> package

> This is the same point. Just for the record, there's a small set of
> packages not based on a Makefile for debian/rules.

I know. I saw the list Russ posted. Not enough to change the
rule. This is a policy directive which has been in place for over 6
years. The packages need to be fixed.

>> d) stands in the way of technical proposals like passing information
>> to the build system on the command line
>> e) prevents people from relying on make semantics for builds.

> The two above points are the same argument. The only proposal I know
> it stands on the way of is the one to list implemented targets with a
> special make invocation which seemed flaky anyway.

What seems flaky to you is unimportant, if indeed it works.

>> The only reason for the bug report seems to be
>> a) because we can
>> b) aesthetics
>> c) profit???

> Not constraining the interface if we don't need to? There's a huge
> difference in possibilities between "any script" and "a Makefile".

I do not agree that there is no need to so constrain it. I have
made the argumen in #88111; please read it. And the current brouhahah
is another reason why the make directive could be used to pass
information to the build process since we do know ./debian/rules is an
executable makefile.

> Yes, we can do it in other ways, such as defining which flags or env
> vars have to be honored, or which files have to be read.

Right. We can re-invent the wheel on our own, in a classic
example of NIH, for absolutely no reason -- apart from not liking a
solution that is already in place.

Not a great idea.

manoj
--
"Pull the trigger and you're garbage." Lady Blue
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, 6 Dec 2007 20:54:29 +0100, Loïc Minier <lool@dooz.org> said:
> I got the feeling it was flaky from the criticism I read on
> debian-policy@ and that it couldn't work for all Makefiles; for example
> someone proposed to "make -f debian/rules -pn | grep '^build-arch:'"
> but this obviously wont fly if this is implemented as a "build-%:"
> rule.
> In fact I have packages with build-% rules, and certainly they
> shouldn't match such checks as they don't implement build-arch.

*Sigh*.

__> cat makefile
build-%:
echo $@
__> make build-foo
echo build-foo
build-foo
__> make build-arch
echo build-arch
build-arch
__> make -pn build-arch | grep '^build-arch'
build-arch:

OK?

> I did; I don't think you can reliably include a Makefile written by
> somebody else with the only constraints of the current policy. It
> would require tons of policies to make this reliable.

But at least it is a starting point.

> Process interfaces draw a much clearer line. I'm sure that as the make
> maintainer you do know how complex a Makefile can get.

You can write FORTRAN in any language, sure. But for the most
part, since we specify the entry points for the make call anyway, the
complexity does not matter, as long as the policy requements for
targets and ordering are met.

> It seems much more easy to usefully standardize the behavior of a
> process (flags / arguments, exit code, env vars, external deps,
> current working directory: sounds sane) than of a Makefile.

With make as a starting point, most of this already present, and
even standardized. Make has a well known interface, it's behaviour is
also defined; we no longer have to reinvent the wheel.

>>> Yes, we can do it in other ways, such as defining which flags or env
>>> vars have to be honored, or which files have to be read.
>> Right. We can re-invent the wheel on our own, in a classic
>> example of NIH, for absolutely no reason -- apart from not liking a
>> solution that is already in place.

> I disagree that it's for no reason; some proposed uses aren't safe and
> we have better replacement proposals (such as using a control field or
> an env var).

I do not think you have made your thesis that the alternatives
are better.

> > Yes, we can do it in other ways, such as defining which flags or env
> > vars have to be honored, or which files have to be read.
> Right. We can re-invent the wheel on our own, in a classic
> example of NIH, for absolutely no reason -- apart from not liking a
> solution that is already in place.

> We already proved the use over many more years of env vars passed to
> debian/rules, arguments passed on the command-line, or of data in
> debian/control. Proposing to use the new channels such as makefile
> inclusion or querying for the implemented rules is looking for trouble
> and discourages other options.

What new channel? This channel (./debian/rules as a Makfile)
has been in practice since around 1996, and in policy as a mUST since
2001 (which was, I think, before you became a DD).

If you think it is new, you must not really have read policy
very well.

> I'm not under the impression that you're still open-minded on the
> topic, and since you're one of our beloved policy maintainers, and make
> maintainer, I don't think it's worth our time to repeat the arguments
> which have already been made.

Ah. Argumentum ad Hominem. "I can't refute Manoj's arguments, so
let us paint him as an irrational bigot". I am going to ignore this as
the logical fallacy that it is.

manoj
--
"Anger makes you smaller, while forgiveness forces you to grow beyond
what you were." -Cherie Carter-Scott
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org