RFD: EAPI specification in ebuilds
 

Hi all,

The way how we currently specify the EAPI in ebuilds has some
problems. For example, there is no sane way to allow usage of features
of a new bash version in a new EAPI. So we are currently stuck with
bash 3.2. Also changes of global scope behaviour, like addition of new
global scope functions (similar to "inherit") are not possible.

These flaws are outlined in GLEP 55 [1]:
| In order to get the EAPI the package manager needs to source the
| ebuild, which itself needs the EAPI in the first place. Otherwise it
| imposes a serious limitation, namely every ebuild, using any of the
| future EAPIs, will have to be source'able by old package managers
| [...]

The council has voted down GLEP 55 more than a year ago, but at the
same time requested that a solution for the mentioned issues should be
found. [2] However, there was no progress since then.

The issue arose again in bug 402167 [3] where several solutions have
been discussed. Below, I try to summarise the possible options
resulting from that discussion.


*** Proposal 1: "Parse the EAPI assignment statement" ***

This first proposal would require that the syntax of the EAPI
assignment statement in ebuilds matches a well defined regular
expression. A scan of the Portage tree shows that the statement only
occurs in the following variations (using EAPI 4 as example):

EAPI=4
EAPI="4"
EAPI='4'

Sometimes this is followed by whitespace or a comment (starting with
a # sign). Also, with very few exceptions the EAPI assignment occurs
within the first few lines of the ebuild. For the vast majority of
ebuilds it is in line 5.

Written in a more formal way, appropriate for a specification:
- Ebuilds must contain at most one EAPI assignment statement.
- It must occur within the first N lines of the ebuild (N=10 and N=30
have been suggested).
- The statement must match the following regular expression (extended
regexp syntax):
^[ ]*EAPI=(['"]?)([A-Za-z0-9._+-]*)1[ ]*(#.*)?$

Note: The first and the third point are already fulfilled by all
ebuilds in the Portage tree. The second point will require very few
ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).

The package manager would determine the EAPI by parsing the assignment
with above regular expression. A sanity check would be added. Citing
Zac Medico in [3]: "The fact that we can compare the probed EAPI to
the actual EAPI variable after the ebuild is sourced seems like a
perfect sanity check. We could easily detect inconsistencies and flag
such ebuilds as invalid, providing a reliable feedback mechanism to
ebuild developers."

This proposal comes in two variants:
1a) The change is applied retroactively for all EAPIs.
1b) It is only applied for EAPI 5 and later (which means that the
result of the EAPI parsing would be discarded for earlier EAPIs).


*** Proposal 2: "EAPI in header comment" ***

A different approach would be to specify the EAPI in a specially
formatted comment in the ebuild's header. No syntax has been suggested
yet, but I believe that the following would work as a specification:
- The EAPI must be declared in a special comment in the first line of
the ebuild's header, as follows:
- The first line of the ebuild must contain the word "ebuild",
followed by whitespace, followed by the EAPI, followed by
end-of-line or whitespace.

Again, the proposal comes in two variants:
2a) It is combined with a one time change of the file extension, like
.ebuild -> .eb.
2b) The usual EAPI assignment statement in the ebuild is still
required, at least for a transition period.

In the 2a case, the EAPI variable could be made read-only in bash
before sourcing the ebuild. In the 2b case, a sanity check similar to
the one mentioned above would be added.


What do you think?

(I really hope for a constructive discussion here. So, if you want
to comment that all of the above proposals suck and GLEP 55 is much
superior, then please open a new thread for it.)

Ulrich


[1] <http://www.gentoo.org/proj/en/glep/glep-0055.html#problem>
[2] <http://www.gentoo.org/proj/en/council/meeting-logs/20100823-summary.txt>
[3] <https://bugs.gentoo.org/show_bug.cgi?id=402167>

RFD: EAPI specification in ebuilds
 

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> (I really hope for a constructive discussion here. So, if you want
> to comment that all of the above proposals suck and GLEP 55 is much
> superior, then please open a new thread for it.)

I think if you want a constructive discussion, then you should ignore
the previous Council's decision and reconsider all the solutions that
have been presented.

--
Ciaran McCreesh

RFD: EAPI specification in ebuilds
 

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> Hi all,
>
> The way how we currently specify the EAPI in ebuilds has some
> problems. For example, there is no sane way to allow usage of features
> of a new bash version in a new EAPI. So we are currently stuck with
> bash 3.2. Also changes of global scope behaviour, like addition of new
> global scope functions (similar to "inherit") are not possible.
>
> These flaws are outlined in GLEP 55 [1]:
> | In order to get the EAPI the package manager needs to source the
> | ebuild, which itself needs the EAPI in the first place. Otherwise it
> | imposes a serious limitation, namely every ebuild, using any of the
> | future EAPIs, will have to be source'able by old package managers
> | [...]
>
> The council has voted down GLEP 55 more than a year ago, but at the
> same time requested that a solution for the mentioned issues should be
> found. [2] However, there was no progress since then.
>
> The issue arose again in bug 402167 [3] where several solutions have
> been discussed. Below, I try to summarise the possible options
> resulting from that discussion.
>
>
> *** Proposal 1: "Parse the EAPI assignment statement" ***
>
> This first proposal would require that the syntax of the EAPI
> assignment statement in ebuilds matches a well defined regular
> expression. A scan of the Portage tree shows that the statement only
> occurs in the following variations (using EAPI 4 as example):
>
> EAPI=4
> EAPI="4"
> EAPI='4'
>
> Sometimes this is followed by whitespace or a comment (starting with
> a # sign). Also, with very few exceptions the EAPI assignment occurs
> within the first few lines of the ebuild. For the vast majority of
> ebuilds it is in line 5.
>
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
> have been suggested).
> - The statement must match the following regular expression (extended
> regexp syntax):
> ^[ ]*EAPI=(['"]?)([A-Za-z0-9._+-]*)1[ ]*(#.*)?$
>
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
>
> The package manager would determine the EAPI by parsing the assignment
> with above regular expression. A sanity check would be added. Citing
> Zac Medico in [3]: "The fact that we can compare the probed EAPI to
> the actual EAPI variable after the ebuild is sourced seems like a
> perfect sanity check. We could easily detect inconsistencies and flag
> such ebuilds as invalid, providing a reliable feedback mechanism to
> ebuild developers."
>
> This proposal comes in two variants:
> 1a) The change is applied retroactively for all EAPIs.
> 1b) It is only applied for EAPI 5 and later (which means that the
> result of the EAPI parsing would be discarded for earlier EAPIs).

+1 for the whole proposal, this seems to be solving the problem and is
the less intrusive change

As i understand it, $PM will need to try the regexp tingy on any ebuild
anyway, guess the EAPI then source the ebuild with the right sourcer
to get the real EAPI and compare it. So, for me, the proposal is
retroactive. The check whether guessed and actual EAPI are the same can
be made a warning, or just ignored, in EAPI<5 and fatal in >=5 though.

A.

RFD: EAPI specification in ebuilds
 

On 7 March 2012 21:07, Alexis Ballier <aballier@gentoo.org> wrote:
> As i understand it, $PM will need to try the regexp tingy on any ebuild
> anyway, guess the EAPI then source the ebuild with the right sourcer
> to get the real EAPI and compare it.

Not exactly... the idea with proposal 2) is that the header comment
becomes the One True Way to set the EAPI, so it wouldn't be "guessing"
as such, and ebuilds wouldn't be allowed to change it during sourcing
any more than they can redefine PV or the like. As mentioned, 2b)
still risks a mismatch between the header and the assignment, but
hopefully that would be temporary and the assignments could be dropped
eventually. (And I'd suggest that the header EAPI is still considered
authoritative if present - the mismatch check should probably be a
hard error, or if not it should generate a warning and use the header
value. There should be minimal if any risk of this changing behaviour
for any existing ebuild, since existing ebuilds almost certainly won't
match the chosen header syntax.)

As for my opinion, I would prefer 2a), or 2b) as a second choice.
IMHO it's much simpler and cleaner to come up with a strict,
well-defined header format than to try and work within (an arbitrarily
restricted version of) bash syntax.

RFD: EAPI specification in ebuilds
 

On 03/07/2012 03:41 PM, Ulrich Mueller wrote:


*** Proposal 2: "EAPI in header comment" ***

A different approach would be to specify the EAPI in a specially
formatted comment in the ebuild's header. No syntax has been suggested
yet, but I believe that the following would work as a specification:
- The EAPI must be declared in a special comment in the first line of
the ebuild's header, as follows:
- The first line of the ebuild must contain the word "ebuild",
followed by whitespace, followed by the EAPI, followed by
end-of-line or whitespace.



Someone suggested using a standard shebang the last time this came up,
and if I remember correctly it was one of the least-disagreeable
solutions proposed. We could of course define our own custom format, but
I think something like,


#!/usr/bin/eapi5

would be perfect if we could hand off the interpretation of the ebuild
to that program. That solves the problem with new bash features, too,
since you could point that command at a specific version.

RFD: EAPI specification in ebuilds
 

On Wed, 2012-03-07 at 21:41 +0100, Ulrich Mueller wrote:
> .ebuild -> .eb

FYI, any Russian speaker is *guaranteed* to read the name ".eb" as a
very common obscenity.

RFD: EAPI specification in ebuilds
 

>>>>> On Wed, 07 Mar 2012, Michael Orlitzky wrote:

> Someone suggested using a standard shebang the last time this came
> up, and if I remember correctly it was one of the least-disagreeable
> solutions proposed. We could of course define our own custom format,
> but I think something like,

> #!/usr/bin/eapi5

> would be perfect if we could hand off the interpretation of the
> ebuild to that program. That solves the problem with new bash
> features, too, since you could point that command at a specific
> version.

Please note that I've formulated the specification in a way that
accounts for this:

>> - The first line of the ebuild must contain the word "ebuild",
>> followed by whitespace, followed by the EAPI, followed by
>> end-of-line or whitespace.

This would also match the line "#!/usr/bin/ebuild 5" (if the magic
word should be "ebuild" or "eapi" can be discussed of course).
So even if we would introduce the shebang only in a later EAPI,
previous portage versions would still recognise the new header.

Ulrich

RFD: EAPI specification in ebuilds
 

On Wed, 07 Mar 2012 17:36:05 -0500
Alexandre Rostovtsev <tetromino@gentoo.org> wrote:

> FYI, any Russian speaker is *guaranteed* to read the name ".eb" as a
> very common obscenity.

In Dutch it means the low tide, and as a verb, it means "becoming low"
or "decreasing" as in the tide or some other fluid.

In English it means something very similar as in the Dutch verb, only
without the tidal reference. Other Germanic languages will probably
have their own variants.

In Crimean Tatar it apparently[1] references "easiness" or "simplicity",
in Hungarian it's the word for "dog", while in Nauruan it means
"country."

But that's not important right now (see also [2]).

The important thing to remember is, the bike shed protects our bicycles
from the elements.


jer


[1] http://en.wiktionary.org/wiki/eb
[2] http://en.wikipedia.org/wiki/bytesex

RFD: EAPI specification in ebuilds
 

On Wed, Mar 7, 2012 at 12:41 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
> Hi all,
>
> The way how we currently specify the EAPI in ebuilds has some
> problems. For example, there is no sane way to allow usage of features
> of a new bash version in a new EAPI. So we are currently stuck with
> bash 3.2. Also changes of global scope behaviour, like addition of new
> global scope functions (similar to "inherit") are not possible.
>
> These flaws are outlined in GLEP 55 [1]:
> | In order to get the EAPI the package manager needs to source the
> | ebuild, which itself needs the EAPI in the first place. Otherwise it
> | imposes a serious limitation, namely every ebuild, using any of the
> | future EAPIs, will have to be source'able by old package managers
> | [...]
>
> The council has voted down GLEP 55 more than a year ago, but at the
> same time requested that a solution for the mentioned issues should be
> found. [2] However, there was no progress since then.
>
> The issue arose again in bug 402167 [3] where several solutions have
> been discussed. Below, I try to summarise the possible options
> resulting from that discussion.
>
>
> *** Proposal 1: "Parse the EAPI assignment statement" ***
>
> This first proposal would require that the syntax of the EAPI
> assignment statement in ebuilds matches a well defined regular
> expression. A scan of the Portage tree shows that the statement only
> occurs in the following variations (using EAPI 4 as example):
>
> * EAPI=4
> * EAPI="4"
> * EAPI='4'
>
> Sometimes this is followed by whitespace or a comment (starting with
> a # sign). Also, with very few exceptions the EAPI assignment occurs
> within the first few lines of the ebuild. For the vast majority of
> ebuilds it is in line 5.
>
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
> *have been suggested).
> - The statement must match the following regular expression (extended
> *regexp syntax):
> *^[ ]*EAPI=(['"]?)([A-Za-z0-9._+-]*)1[ ]*(#.*)?$
>
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
>
> The package manager would determine the EAPI by parsing the assignment
> with above regular expression. A sanity check would be added. Citing
> Zac Medico in [3]: "The fact that we can compare the probed EAPI to
> the actual EAPI variable after the ebuild is sourced seems like a
> perfect sanity check. We could easily detect inconsistencies and flag
> such ebuilds as invalid, providing a reliable feedback mechanism to
> ebuild developers."
>
> This proposal comes in two variants:
> 1a) The change is applied retroactively for all EAPIs.
> 1b) It is only applied for EAPI 5 and later (which means that the
> * *result of the EAPI parsing would be discarded for earlier EAPIs).

I don't like this idea because the sane way should be easy and straightforward.
Mixing a constant declaration with bash assignment just confuses users
who think the assignment is full bash when in fact it is not.

EAPI=$(somefunc)
EAPI=${SOMEVAR%%-*}
and so forth all don't meet the regex (and would be flagged invalid.)
However a naive author might think they work.

I don't think any naive author would think either would work in a comment

# EAPI=$(somefunc)
# EAPI=${SOEMVAR%%-*}

Bash doesn't parse comments, so confusion is unlikely.

>
>
> *** Proposal 2: "EAPI in header comment" ***
>
> A different approach would be to specify the EAPI in a specially
> formatted comment in the ebuild's header. No syntax has been suggested
> yet, but I believe that the following would work as a specification:
> - The EAPI must be declared in a special comment in the first line of
> *the ebuild's header, as follows:
> - The first line of the ebuild must contain the word "ebuild",
> *followed by whitespace, followed by the EAPI, followed by
> *end-of-line or whitespace.
>
> Again, the proposal comes in two variants:
> 2a) It is combined with a one time change of the file extension, like
> * *.ebuild -> .eb.
> 2b) The usual EAPI assignment statement in the ebuild is still
> * *required, at least for a transition period.
>
> In the 2a case, the EAPI variable could be made read-only in bash
> before sourcing the ebuild. In the 2b case, a sanity check similar to
> the one mentioned above would be added.
>
>
> What do you think?

Overloading is bad.

There is no real difference between:
#!/usr/bin/ebuild --eapi 5
# EAPI=5

The problem is that the former is also the way to specify how to
execute the ebuild; so unless you plan to make ebuilds executable and
having /usr/bin/ebuild provide the ebuild environment, using that
syntax is confusing to users.

>
> (I really hope for a constructive discussion here. So, if you want
> to comment that all of the above proposals suck and GLEP 55 is much
> superior, then please open a new thread for it.)

You don't mention voting on glep 55 again; is there a reason why not?

>
> Ulrich
>
>
> [1] <http://www.gentoo.org/proj/en/glep/glep-0055.html#problem>
> [2] <http://www.gentoo.org/proj/en/council/meeting-logs/20100823-summary.txt>
> [3] <https://bugs.gentoo.org/show_bug.cgi?id=402167>
>

RFD: EAPI specification in ebuilds
 

>>>>> On Wed, 7 Mar 2012, Alec Warner wrote:

>> *** Proposal 1: "Parse the EAPI assignment statement" ***
>> [...]

> I don't like this idea because the sane way should be easy and
> straightforward. Mixing a constant declaration with bash assignment
> just confuses users who think the assignment is full bash when in
> fact it is not.

> EAPI=$(somefunc)
> EAPI=${SOMEVAR%%-*}
> and so forth all don't meet the regex (and would be flagged
> invalid.) However a naive author might think they work.

Such constructs also cannot be used with any of the other proposed
solutions. And in fact, nobody is using such things in practice.
_All_ ebuilds in the Portage tree can be successfully parsed with the
regexp proposed.

The obvious sanity check, i.e. comparing the EAPI obtained by parsing
and by sourcing, could be added to repoman, which would prevent such
non-conforming ebuilds from being committed to the tree.

>> *** Proposal 2: "EAPI in header comment" ***
>> [...]

> Overloading is bad.

> There is no real difference between:
> #!/usr/bin/ebuild --eapi 5
> # EAPI=5

> The problem is that the former is also the way to specify how to
> execute the ebuild; so unless you plan to make ebuilds executable and
> having /usr/bin/ebuild provide the ebuild environment, using that
> syntax is confusing to users.

I agree with this point.

Many file formats are identifying themselves with a magic token (as
it is used by sys-apps/file), but it is not necessarily a shebang.

Ulrich