How does...?
 

Hello again,

I'm trying to understand how Koji does things in order to migrate from our
previous home-made build system with svn repositories to Koji. Most of our
stuff is in svn as source files, not tarballs, so I'll have to implement a
"make sources". I was looking in the Fedora git repositories, but couldn't
see how this is actually done. I can see how it _used_ to be done: with a
common package containing a Makefile.common. However that doesn't seem to
be available any more.

I've found fedora-packager and fedpkg; there was a thread here in 2010
(http://www.mail-archive.com/buildsys@lists.fedoraproject.org/msg00619.html)
which suggested that they configured Koji to issue the command to get the
sources. However, that's "allowed_scms" in kojid.conf, and I can't see
anything in either package now which modifies that.

So basically I'd really like to know the steps that Fedora's Koji goes
through to build packages like anaconda - which has an old Makefile, and has
no URL to say where to find the sources.


Moray.
"To err is human; to purr, feline."






--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

I still use the old "make sources" way. Pretty much copied from the Fedora set up, with the "lookaside" repository for the files.
Here's how I understood it, please correct me if I'm wrong:

Koji checks out the package subdir and executes "make sources" there.*The Makefile within the subdir of the package in svn references a "Makefile.common" that lives in "trunk/common". The Makefile retrieves the "Makefile.common" from svn and loads it. Makefile.common then retrieves the sources file and the sources.

To download them it uses the REPOSITORY variable, i.e. the svn host. It fetches the sources like this:http://REPOSTITORY_HOST/repo/pkgs/<packagename>/<source file name>/<md5sum of source file>/<source file name>. If the md5sum matches, "make sources" returns and koji proceeds with building.



As Jesse wrote back then, if you don't want to do it that way, you're free to specify any other command that will download the sources. For me it was a good start without getting too much into fedpkg stuff.


Andreas.

On Fri, Apr 20, 2012 at 18:34, Moray Henderson <Moray.Henderson@ict-software.org> wrote:


Hello again,



I'm trying to understand how Koji does things in order to migrate from our

previous home-made build system with svn repositories to Koji. *Most of our

stuff is in svn as source files, not tarballs, so I'll have to implement a

"make sources". *I was looking in the Fedora git repositories, but couldn't

see how this is actually done. *I can see how it _used_ to be done: with a

common package containing a Makefile.common. *However that doesn't seem to

be available any more.



I've found fedora-packager and fedpkg; there was a thread here in 2010

(http://www.mail-archive.com/buildsys@lists.fedoraproject.org/msg00619.html)

which suggested that they configured Koji to issue the command to get the

sources. *However, that's "allowed_scms" in kojid.conf, and I can't see

anything in either package now which modifies that.



So basically I'd really like to know the steps that Fedora's Koji goes

through to build packages like anaconda - which has an old Makefile, and has

no URL to say where to find the sources.





Moray.

"To err is human; to purr, feline."













--

buildsys mailing list

buildsys@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/buildsys

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

Here's the Makefile.common that I use.
Andreas








--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

Hello Moray,

As Andreas pointed out there are a lot of pieces involved in getting
koji to build from git or hg or other revision control systems. I'll
try to explain the process from my POV as I've now done it a few
times. The project I work on is called GoOSe Linux, it's a similar
project to what CentOS is doing. It's possible I'm going into too much
information, but I figured it could not hurt to give you the pieces,
included the makefile and makefile.common parts.

Essentially, we have all of our rpms separated into two basic
components. First, we have a git repository for all of the rpms we
wish to build. They can be found at
http://github.com/gooselinux/<repo>. These consist of the spec, patch
and Makefiles along with a file called sources. This is exactly
similar to the method Andreas described in the previous mail. The
other part is the lookaside cache which holds the actual upstream
source code, usually in a tarball or zip file. These are all of the
pieces you need to make koji work properly.

Assuming you have your koji system setup and the package tagged, you
just have to tell koji to build your package. This can be done simply
by running 'koji build gl6-alpha
git://github.com/gooselinux/anaconda.git#HEAD' (your koji build line
will vary). As other discussions have probably explained, there's a
lot of magic that happens here, so I'll go through it step by step.

What happens next is that koji sets up a mock build environment to
build the source rpm. Once the mock environment is initialized, make
sources is called. Koji then takes the url you provided and checks it
out into the mock build environment. Then, the lookaside cache is
downloaded. You should have a look at our Makefile.common at
https://github.com/gooselinux/common. There is also a Makefile for
each rpm we have koji build, using anaconda as an example, you can
find the Makefile at https://github.com/gooselinux/anaconda. The
Makefile downloads the tarball from the lookaside cache and verifies
the sha256sum from the 'sources' file in the anaconda git repository
previously mentioned. If all of that completes successfully, then mock
builds the source rpm.

After building the source rpm, koji then creates the mock
environment(s) to build the binary rpm(s). The rpm(s) is/are built
from the source rpm (SRPM) built in the previous explanation.

A couple things you mentioned about kojid.conf and the allowed_scms
variable. I've attached a copy of my kojid.conf from one of the
builders to help you along your way. Additionally, Andreas'
Makefile.common is pretty long and does a lot of things, it's likely
got some very nice checking and additional things that the goose
Makefile.common doesn't. Ours is rather simple and just does the bare
minimum to make things work.

I sure hope this explanation and files has helped you on your way to
using Koji to build rpms!

Cheers,

Clint

On Fri, Apr 20, 2012 at 1:47 PM, Andreas Mack <andreas.mack@gmail.com> wrote:
> Here's the Makefile.common that I use.
>
> Andreas
>
>
>>
>
>
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

On Fri, 2012-04-20 at 17:34 +0100, Moray Henderson wrote:

> I've found fedora-packager and fedpkg; there was a thread here in 2010
> (http://www.mail-archive.com/buildsys@lists.fedoraproject.org/msg00619.html)
> which suggested that they configured Koji to issue the command to get the
> sources. However, that's "allowed_scms" in kojid.conf, and I can't see
> anything in either package now which modifies that.

I wouldn't assume that the baseline configuration of the koji package in
Fedora necessarily matches how Fedora's koji instance is deployed.

> So basically I'd really like to know the steps that Fedora's Koji goes
> through to build packages like anaconda - which has an old Makefile, and has
> no URL to say where to find the sources.

Well, the way you do this as a fedpkg user is just run "fedpkg sources",
which grabs tarballs from the lookaside server based on the names and
md5sums in the 'sources' file in the checked-out package. Which is how
"make sources" used to work, too.

From a quick look at the kojid source I'd assume you'd want to override
the 'source_cmd' attribute of the SCM to do whatever's appropriate.

- ajax
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

On 04/20/2012 12:34 PM, Moray Henderson wrote:

So basically I'd really like to know the steps that Fedora's Koji goes
through to build packages like anaconda - which has an old Makefile, and has
no URL to say where to find the sources.


The allowed_scms option in kojid specifies which scms can be used and
how kojid should use them. The format of the option is:


"""
a space-separated list of host:repository[:use_common[:source_cmd]]
tuples. Incorrectly-formatted tuples will be ignored.


If use_common is not present, kojid will attempt to checkout a common/
directory from the repository. If use_common is set to no, off, false,
or 0, it will not attempt to checkout a common/ directory.


source_cmd is a shell command (args separated with commas instead of
spaces) to run before building the srpm. It is generally used to
retrieve source files from a remote location. If no source_cmd is
specified, "make sources" is run by default.

"""

So...
- if you were using an old dist-cvs setup, you'd set use_common to yes
and leave source_cmd as the default.
- with a dist-git setup, you'd set use_common to no and source_cmd to
fedpkg,sources
- for simpler setups, you might simply set use_common to no, leave
source_cmd as the default, and ensure that each package to be built
includes a Makefile with a sources target that does the right thing.


Which of these options is best depends greatly on your situation.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

How does...?
 

Thanks everyone - your replies have been very helpful.


Moray.
"To err is human; to purr, feline."


OM International Limited - Unit B Clifford Court, Cooper Way - Carlisle CA3 0JG - United Kingdom
Charity reg no: 1112655 - Company reg no: 5649412 (England and Wales)

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys