FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Packaging

 
 
LinkBack Thread Tools
 
Old 02-16-2012, 09:50 PM
Jon Masters
 
Default Scripting language dependencies

Greetings,

I hope everyone is having a good day. Here in the Fedora ARM project,
we're having a great time overall, but we've run up against a familiar
foe in our preparation for Fedora 17 and rawhide: superfluous deps in
SPEC files. Today's example is a shell-escaped call to Ruby just to
determine some gcc flags for an unrelated package build, but that's just
today's example. There are many others, so some standard would help.

Why is this a problem? The most obvious issue is the superfluous
dependencies, pulling in ruby (and all of its deps), for example. But
the bigger problem for the distribution as a whole is that we are
becoming less bootstrappable[0] over time, and we are suffering from
dependency creep that makes life difficult for secondary architectures
(or even for the primary architecture if a bunch of rebuilds were to
fail and there were no deps to satisfy a circular path).

Eventually, the issue of dependency creep is going to bite the broader
distribution. In the meantime, can we please impose some kind of rule
surrounding which superfluous build deps are allowed? I'm not advocating
for the removal of Ruby, Lua, and so on and a return to old fashioned
shell, but I am suggesting that we cap where we are and try to avoid
introducing any more bizzare build dependencies in the future. For
example, say a package grew a SPEC file need for a call to an INTERCAL
script just to determine gcc flags? It's not really all that different
from calling Ruby to expand gcc flags and maybe I happen to like using
INTERCAL over Ruby? Or why not have a hardware compiled state machine
implement some custom logic to determine gcc flags? Maybe I find that
easier than writing some good old fashioned shell, etc.

In all seriousness, can we please have some direction? If this is
already implied or codified, could someone remind packagers to refrain
from adding build dependencies like this? I'd recommend that, in
general, if a one line shell escape can be used, that ought to be the
preference over adding a one-line build dep on $random scripting lang.

Thanks!

Jon.

[0] Rebuilding the entire distribution from first principles, using
nothing more than a toolchain to begin the process. We did this for
Fedora 15 on ARMv7. I eventually would like it that Fedora be able to
bootstrap itself on any architecture trivially, and I know Debian (and
other distributions) are similarly keen to return to that world.


--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 01:16 PM
Bill Nottingham
 
Default Scripting language dependencies

Jon Masters (jcm@redhat.com) said:
> I hope everyone is having a good day. Here in the Fedora ARM project,
> we're having a great time overall, but we've run up against a familiar
> foe in our preparation for Fedora 17 and rawhide: superfluous deps in
> SPEC files. Today's example is a shell-escaped call to Ruby just to
> determine some gcc flags for an unrelated package build, but that's just
> today's example. There are many others, so some standard would help.

OK, so how about:

...
Scripting inside of spec files

Inside of a spec file, sometimes it is necessary to call a programming or
scripting language during %setup, %build, or %install. In Fedora, spec
files may in general only use the following languages for this purpose:

1. Python
2. Perl
3. awk/sed

Also, if your package already BuildRequires a specific scripting language
(such as Ruby, or Tcl) as part of its normal compile process, it may also be
called from the spec file.

...

Bill
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging��y�{*��^���aj�!� �[��ƫrb��'������{�j�!� ޖ+ޯ'Z��^����z����z+�� W^j�!�)����x���o+6��ڶ*l �0���ki�������� �zw�jV���܆X��+�'uG#��M4 �����{�j�!��k�^����z ���z+������{��� ����܆X��+������o(��]y�܆X��+���a#>'C�Z�^om5�u MI4^q���h��]y�܆X��+���W^j�!�)����x �������{������"� 13������r���܆X��+�Z ^om5�u�MI4^q�ԏ��� ��W(���� ��m=��"������m���+� � ��W(�)�&�ק�+r��x��*x�|� {����Ǣ��
 
Old 02-17-2012, 02:10 PM
Tom Lane
 
Default Scripting language dependencies

Bill Nottingham <notting@redhat.com> writes:
> OK, so how about:

> Scripting inside of spec files

> Inside of a spec file, sometimes it is necessary to call a programming or
> scripting language during %setup, %build, or %install. In Fedora, spec
> files may in general only use the following languages for this purpose:

> 1. Python
> 2. Perl
> 3. awk/sed

> Also, if your package already BuildRequires a specific scripting language
> (such as Ruby, or Tcl) as part of its normal compile process, it may also be
> called from the spec file.

Couple of questions:

1. Which of the above alternatives demand a BuildRequires for the
language? (Is BuildRequires the right thing for calls that aren't in
%build?)

2. Can we put even stricter limits on scripting language calls from
macro definitions? As an example, the specfile for python-psycopg2
contains

%{expand: %%define py3ver %(python3 -c 'import sys;print(sys.version[0:3])')}

which I find to be a perpetual annoyance because I don't have python3
installed on my system, and so *any* tool that parses the specfile burps
out an error message, for example pretty nearly any fedpkg command
complains about that. In the particular case, %py3ver isn't used
anywhere that's critical except in an actual build, so things work,
but it's unsightly as can be. I have seen specfiles in other people's
packages where specfile parsing fails entirely unless you have
$randompackage installed, and that seems bad to me.

Now, just instituting a policy edict against this isn't going to fix it,
and if I knew how to get rid of the python-psycopg2 usage I would have
done so already. But if we're going to have anything about scripting
languages in the policy then I'd like to have some guidance about cases
like this.

regards, tom lane
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 03:13 PM
Toshio Kuratomi
 
Default Scripting language dependencies

On Fri, Feb 17, 2012 at 09:16:40AM -0500, Bill Nottingham wrote:
> Jon Masters (jcm@redhat.com) said:
> > I hope everyone is having a good day. Here in the Fedora ARM project,
> > we're having a great time overall, but we've run up against a familiar
> > foe in our preparation for Fedora 17 and rawhide: superfluous deps in
> > SPEC files. Today's example is a shell-escaped call to Ruby just to
> > determine some gcc flags for an unrelated package build, but that's just
> > today's example. There are many others, so some standard would help.
>
This isn't codified in current Guidelines that I know of. I do recall one
thread on it from the very early days of Fedora (possibly before the FPC).

> OK, so how about:
>
> ...
> Scripting inside of spec files
>
> Inside of a spec file, sometimes it is necessary to call a programming or
> scripting language during %setup, %build, or %install. In Fedora, spec
> files may in general only use the following languages for this purpose:
>
> 1. Python
> 2. Perl
> 3. awk/sed
>
> Also, if your package already BuildRequires a specific scripting language
> (such as Ruby, or Tcl) as part of its normal compile process, it may also be
> called from the spec file.
>
That seems about right. Maybe be a little looser and allow using languages
that are either BuildRequires or Requires already.

For instance, take a language that doesn't byte compile itself. If you had
a package where upstream consists of a single file that you then have to cp
to the right filesystem location to become a module for that language it
wouldn't need the interpreter to compile. However, the best way to
determine the filesystem location might be something like::
"foolanguage -e 'print libpath'"

It seems appropriate to use that in a spec file.

Also, #3 might be better if it was a little more vague::

"Standard programs used in shell programming, for instance awk and sed"

Another clarificaation might be to make the initial paragraph a little more
specific:

"Sometimes it is necessary to write a short script (perhaps a one-liner)
that's executed in %setup, %build, or %install to get some information
about the build environment. In Fedora, spec files may in general only use
the following languages for this purpose: [...]"

-Toshio
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 03:43 PM
Toshio Kuratomi
 
Default Scripting language dependencies

On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
> Bill Nottingham <notting@redhat.com> writes:
> > OK, so how about:
>
> > Scripting inside of spec files
>
> > Inside of a spec file, sometimes it is necessary to call a programming or
> > scripting language during %setup, %build, or %install. In Fedora, spec
> > files may in general only use the following languages for this purpose:
>
> > 1. Python
> > 2. Perl
> > 3. awk/sed
>
> > Also, if your package already BuildRequires a specific scripting language
> > (such as Ruby, or Tcl) as part of its normal compile process, it may also be
> > called from the spec file.
>
> Couple of questions:
>
> 1. Which of the above alternatives demand a BuildRequires for the
> language? (Is BuildRequires the right thing for calls that aren't in
> %build?)
>
Technically, only python. For good practice I would say both perl and
python. I would never say awk/sed.

Which reminds me... maybe we need to specify gawk rather than awk... if
someone wrote something that worked only with mawk we wouldn't like that
either.

BuildRequires is correct for %setup, %build, %install, and %check... the
sections that are run when building a package. %pre/%post/etc would use
a form of Requires([pre,post,etc]): instead as they are run on the end
user's system at package install time.

> 2. Can we put even stricter limits on scripting language calls from
> macro definitions? As an example, the specfile for python-psycopg2
> contains
>
> %{expand: %%define py3ver %(python3 -c 'import sys;print(sys.version[0:3])')}
>
> which I find to be a perpetual annoyance because I don't have python3
> installed on my system, and so *any* tool that parses the specfile burps
> out an error message, for example pretty nearly any fedpkg command
>
The very general approach of using a scripting language that the package (or
a subpackage) is building for seems valid. So I don't think that this
should be restricted here.

That macro could be better, though. For instance, if you use this:
%{!?py3ver: %global py3ver %(%{__python3} -c 'import sys; print(sys.version[0:3])')}

You won't get the errors that are coming from that %{expand} line.

However, that won't prevent you from having other errors/warnings like this:
sh: /usr/bin/python3: No such file or directory

Without python3 installed, macros in the spec file can't be expanded
correctly (because their definitions depend on python3). The spec file is
BuildRequireing python3 so it shouldn't be expected that you can operate on
the spec file without python3 installed.

-Toshio
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 05:23 PM
Paul Howarth
 
Default Scripting language dependencies

On Fri, 17 Feb 2012 08:43:19 -0800
Toshio Kuratomi <a.badger@gmail.com> wrote:

> On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
> > Bill Nottingham <notting@redhat.com> writes:
> > > OK, so how about:
> >
> > > Scripting inside of spec files
> >
> > > Inside of a spec file, sometimes it is necessary to call a
> > > programming or scripting language during %setup, %build, or
> > > %install. In Fedora, spec files may in general only use the
> > > following languages for this purpose:
> >
> > > 1. Python
> > > 2. Perl
> > > 3. awk/sed
> >
> > > Also, if your package already BuildRequires a specific scripting
> > > language (such as Ruby, or Tcl) as part of its normal compile
> > > process, it may also be called from the spec file.
> >
> > Couple of questions:
> >
> > 1. Which of the above alternatives demand a BuildRequires for the
> > language? (Is BuildRequires the right thing for calls that aren't
> > in %build?)
> >
> Technically, only python. For good practice I would say both perl and
> python. I would never say awk/sed.
>
> Which reminds me... maybe we need to specify gawk rather than awk...
> if someone wrote something that worked only with mawk we wouldn't
> like that either.
>
> BuildRequires is correct for %setup, %build, %install, and %check...
> the sections that are run when building a package. %pre/%post/etc
> would use a form of Requires([pre,post,etc]): instead as they are run
> on the end user's system at package install time.
>
> > 2. Can we put even stricter limits on scripting language calls from
> > macro definitions? As an example, the specfile for python-psycopg2
> > contains
> >
> > %{expand: %%define py3ver %(python3 -c 'import
> > sys;print(sys.version[0:3])')}
> >
> > which I find to be a perpetual annoyance because I don't have
> > python3 installed on my system, and so *any* tool that parses the
> > specfile burps out an error message, for example pretty nearly any
> > fedpkg command
> >
> The very general approach of using a scripting language that the
> package (or a subpackage) is building for seems valid. So I don't
> think that this should be restricted here.
>
> That macro could be better, though. For instance, if you use this:
> %{!?py3ver: %global py3ver %(%{__python3} -c 'import sys;
> print(sys.version[0:3])')}
>
> You won't get the errors that are coming from that %{expand} line.
>
> However, that won't prevent you from having other errors/warnings
> like this: sh: /usr/bin/python3: No such file or directory

You could get rid of those by using:

%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys;
print(sys.version[0:3])' 2>/dev/null)}

And sometimes you might even prefer:

%{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys;
print(sys.version[0:3])' 2>/dev/null || echo 3.0)}

That would give a result that "looked right" in the absence of python3,
giving other bits of the spec that depend on that definition a better
chance of working as expected.

> Without python3 installed, macros in the spec file can't be expanded
> correctly (because their definitions depend on python3). The spec
> file is BuildRequireing python3 so it shouldn't be expected that you
> can operate on the spec file without python3 installed.

I'd prefer to see specs a bit more robust so that for instance you
could run "spectool" on them to download upstream sources and then do a
mockbuild, which wouldn't require python3 or whatever to be installed
on the build host.

Paul.
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 06:33 PM
Toshio Kuratomi
 
Default Scripting language dependencies

On Fri, Feb 17, 2012 at 06:23:25PM +0000, Paul Howarth wrote:
> Toshio Kuratomi <a.badger@gmail.com> wrote:
> >
> > However, that won't prevent you from having other errors/warnings
> > like this: sh: /usr/bin/python3: No such file or directory
>
> You could get rid of those by using:
>
> %{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys;
> print(sys.version[0:3])' 2>/dev/null)}
>

> And sometimes you might even prefer:
>
> %{!?py3ver: %global py3ver %(%{?__python3} -c 'import sys;
> print(sys.version[0:3])' 2>/dev/null || echo 3.0)}
>
> That would give a result that "looked right" in the absence of python3,
> giving other bits of the spec that depend on that definition a better
> chance of working as expected.
>
> > Without python3 installed, macros in the spec file can't be expanded
> > correctly (because their definitions depend on python3). The spec
> > file is BuildRequireing python3 so it shouldn't be expected that you
> > can operate on the spec file without python3 installed.
>
> I'd prefer to see specs a bit more robust so that for instance you
> could run "spectool" on them to download upstream sources and then do a
> mockbuild, which wouldn't require python3 or whatever to be installed
> on the build host.
>
I tested this with spectool -g and python-psycopg2 and all variants of that
py3ver line work.(without even a warning). So spectool isn't really an
issue.

-Toshio
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 07:19 PM
Tom Lane
 
Default Scripting language dependencies

Paul Howarth <paul@city-fan.org> writes:
> On Fri, 17 Feb 2012 08:43:19 -0800
> Toshio Kuratomi <a.badger@gmail.com> wrote:
>> On Fri, Feb 17, 2012 at 10:10:48AM -0500, Tom Lane wrote:
>>> [ whinging about python-psycopg2 ]

>> Without python3 installed, macros in the spec file can't be expanded
>> correctly (because their definitions depend on python3). The spec
>> file is BuildRequireing python3 so it shouldn't be expected that you
>> can operate on the spec file without python3 installed.

> I'd prefer to see specs a bit more robust so that for instance you
> could run "spectool" on them to download upstream sources and then do a
> mockbuild, which wouldn't require python3 or whatever to be installed
> on the build host.

Precisely. There are *lots* of situations where we expect to be able
to parse specfiles without necessarily having all their buildreqs
installed (for the most obvious case: to find out what BRs are needed).
I think one of the goals of this guideline should be to prevent creep
in the set of packages that have to be present before tools like fedpkg
will operate on a specfile without complaint.

regards, tom lane
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 07:23 PM
Tom Lane
 
Default Scripting language dependencies

Toshio Kuratomi <a.badger@gmail.com> writes:
> I tested this with spectool -g and python-psycopg2 and all variants of that
> py3ver line work.(without even a warning). So spectool isn't really an
> issue.

I don't know what spectool is, but I can demonstrate what I see every
time I work with this package:

[tgl@rh3 ~]$ cd f-pkg/python-psycopg2/master/
[tgl@rh3 master]$ ls
psycopg2-2.4.4.tar.gz python-psycopg2.spec sources
[tgl@rh3 master]$ fedpkg srpm
sh: python3: command not found
error: Macro %py3ver has empty body
sh: python3: command not found
error: Macro %py3ver has empty body

sh: python3: command not found
error: Macro %py3ver has empty body
Wrote: /home/tgl/f-pkg/python-psycopg2/master/python-psycopg2-2.4.4-1.fc18.src.rpm

The generated SRPM is fine, and I can (and do) test it in mock, but
those messages would certainly scare anybody not familiar with the
package. I've seen similar behavior with other packages. I'd like
this guideline to show how this sort of thing can be avoided.

regards, tom lane
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 
Old 02-17-2012, 07:39 PM
Toshio Kuratomi
 
Default Scripting language dependencies

On Fri, Feb 17, 2012 at 03:23:36PM -0500, Tom Lane wrote:
> Toshio Kuratomi <a.badger@gmail.com> writes:
> > I tested this with spectool -g and python-psycopg2 and all variants of that
> > py3ver line work.(without even a warning). So spectool isn't really an
> > issue.
>
> I don't know what spectool is, but I can demonstrate what I see every
> time I work with this package:
>
Paul's concern was about spectool -g python-psycopg2.spec doing the right
thing.

I checked, and it does.

> [tgl@rh3 ~]$ cd f-pkg/python-psycopg2/master/
> [tgl@rh3 master]$ ls
> psycopg2-2.4.4.tar.gz python-psycopg2.spec sources
> [tgl@rh3 master]$ fedpkg srpm
> sh: python3: command not found
> error: Macro %py3ver has empty body
> sh: python3: command not found
> error: Macro %py3ver has empty body
>
> sh: python3: command not found
> error: Macro %py3ver has empty body
> Wrote: /home/tgl/f-pkg/python-psycopg2/master/python-psycopg2-2.4.4-1.fc18.src.rpm
>
> The generated SRPM is fine, and I can (and do) test it in mock, but
> those messages would certainly scare anybody not familiar with the
> package. I've seen similar behavior with other packages. I'd like
> this guideline to show how this sort of thing can be avoided.
>
I would keep the concerns separate. The original post was about not having
extraneous deps simply because someone is more comfortable writing a routine
check for information in $RANDOM_UNRELATED_LANGUAGE rather than shell.

This concern is about making things that you actually must BuildRequire in
order to build the package not display warnings when the BuildRequires are
not satisfied.

They're very different. And I'm not sure that I would vote the same for
each of those.

-Toshio
--
packaging mailing list
packaging@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
 

Thread Tools




All times are GMT. The time now is 11:59 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org