Hello,
I'm trying to get our koji instance faster; since we build a lot of
packages, we'd like to reduce the latency.

There're some things that usually make mock faster: the options I
tinkered with and seem to improve performance are root_cache,
yum_cache and tmpfs.

All of those are disabled by default when generating mock config
through koji, and there seems no way to override them:

https://fedorahosted.org/koji/browser/koji/__init__.py#L1276


So here come the questions:
1) Am i missing anything?
2) are all those options intentionally locked out?
3) would a patch which let them be configured be accepted into your git repo?
4) any other way to speedup mock?

Thanks!


--
Alan Franzoni
--
contact me at public@[mysurname].eu
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On Wed, Nov 24, 2010 at 9:46 AM, Alan Franzoni <mailing@franzoni.eu> wrote:
> So here come the questions:
> 1) Am i missing anything?

Nope

> 2) are all those options intentionally locked out?

Yes, the entire point of koji is reproducible builds. Allowing a
binary blob tarball to be extracted into your buildroot leaves you
(potentially) not knowing what went into that buildroot, therefore you
can't make the database entries of exactly what was in the buildroot.

> 3) would a patch which let them be configured be accepted into your git repo?

Not a koji developer, but I sure hope not

> 4) any other way to speedup mock?

I'll let someone else answer here, I find the times quite acceptable
(~45 seconds to populate a buildroot, which is the only thing that
root_cache would speed up).
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On Wed, Nov 24, 2010 at 5:47 PM, Jon Stanley <jonstanley@gmail.com> wrote:
>> 2) are all those options intentionally locked out?
>
> Yes, the entire point of koji is reproducible builds. Allowing a
> binary blob tarball to be extracted into your buildroot leaves you
> (potentially) not knowing what went into that buildroot, therefore you
> can't make the database entries of exactly what was in the buildroot.

Usually the root cache is just a cache containing some packages. I
realize this doesn't seem to work with koji because since every chroot
is new, there's no point in caching.

>> 3) would a patch which let them be configured be accepted into your git repo?
>
> Not a koji developer, but I sure hope not

I'd like to have them optionally configurable, not enabled by default :P

What about tmpfs? That shouldn't interfere with build reproducibility.


--
Alan Franzoni
--
contact me at public@[mysurname].eu
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On Wed, 2010-11-24 at 15:46 +0100, Alan Franzoni wrote:
> Hello,
> I'm trying to get our koji instance faster; since we build a lot of
> packages, we'd like to reduce the latency.

You might want to look at this rpm patch:

commit 2fa98b9e6ab00ee0e8200b405088130cbd626a32
Author: Panu Matilainen <pmatilai@redhat.com>
Date: Wed Oct 20 15:11:20 2010 +0300

Lift fsync enable/disable configuration handling to rpmdb layer
- Add a fsync disable/enable function in the backend, call from
rpmdb layer: on initial database creation (including database
rebuild), it's "safe enough" to disable fsync as there's no former
data to protect. Considerably speeds up initial install to eg chroots.
- Always re-enable fsync when closing a read-write database - at that
point there /is/ data to protect.

...this should help the no cached mock case quite a lot.

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On Wed, Nov 24, 2010 at 03:46:31PM +0100, Alan Franzoni wrote:
> Hello,
> I'm trying to get our koji instance faster; since we build a lot of
> packages, we'd like to reduce the latency.

I am patching in squid as documented on the Fedora wiki pages and
add a "proxy=" line into the yum configuration. That part could
also need a koji configuration option and the koji source already has
comments that this should be done better at some point...

squid probably only helps if the builders are not all at the same
location, which is probably the preferred setup.

regards,

Florian La Roche

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On 11/24/2010 09:46 AM, Alan Franzoni wrote:
> Hello,
> I'm trying to get our koji instance faster; since we build a lot of
> packages, we'd like to reduce the latency.
>
> There're some things that usually make mock faster: the options I
> tinkered with and seem to improve performance are root_cache,
> yum_cache and tmpfs.
>
> All of those are disabled by default when generating mock config
> through koji, and there seems no way to override them:
>
> https://fedorahosted.org/koji/browser/koji/__init__.py#L1276

As Jon pointed out, koji is rather paranoid about reproducibility and
avoids certain mock features for that reason.

However, I should point out that mock does support a "site-wide
configuration file" (/etc/mock/site-defaults.cfg). The default site
config for mock does nothing, but you could conceivably set options
there on your builders. A few caveats though:
1) this may be playing with fire a bit
2) you're likely going to damage/break reproducibility
3) you'll want to keep such settings consistent across all your builders.

> So here come the questions:
> 1) Am i missing anything?
> 2) are all those options intentionally locked out?
> 3) would a patch which let them be configured be accepted into your git repo?
> 4) any other way to speedup mock?

As long as you're going down the path of non-reproducibility, you could
also enable the mock ccache plugin in the site config.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On Fri, Dec 3, 2010 at 9:56 PM, Mike McLean <mikem@redhat.com> wrote:
> However, I should point out that mock does support a "site-wide
> configuration file" (/etc/mock/site-defaults.cfg). The default site
> config for mock does nothing, but you could conceivably set options
> there on your builders. A few caveats though:
> *1) this may be playing with fire a bit
> *2) you're likely going to damage/break reproducibility
> *3) you'll want to keep such settings consistent across all your builders.

Sorry for the late response.

AFAIK, kojid overrides mock site-defaults and turns off ccache,
yum_cache and root_cache:

https://fedorahosted.org/koji/browser/koji/__init__.py#L1276

Btw, instead of fiddling with koji-builder internals, we decided to
manually build via mock and then import the packages on koji; this
lets use to do whatever we like with build-time mock parameters and
still use koji as a repository backend, and we're quite satisfied with
the results.



--
Alan Franzoni
--
contact me at public@[mysurname].eu
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

On 12/15/2010 11:28 AM, Alan Franzoni wrote:
> AFAIK, kojid overrides mock site-defaults and turns off ccache,
> yum_cache and root_cache:
>
> https://fedorahosted.org/koji/browser/koji/__init__.py#L1276

Ah, you're right. Sorry for the misinformation.

> Btw, instead of fiddling with koji-builder internals, we decided to
> manually build via mock and then import the packages on koji; this
> lets use to do whatever we like with build-time mock parameters and
> still use koji as a repository backend, and we're quite satisfied with
> the results.

That's always an option, though you lose buildroot content tracking and
task queuing.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys