FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 11-05-2008, 08:55 AM
Dan Hork
 
Default caching packages on koji builder

Hello,

is it possible to somehow cache the packages that are going from koji
hub to the builder to create a build root? There is no NFS connection
between the builder and hub in s390's koji and so each build requires to
download 100 MB via XMLRPC(?) and this is slow. Will squid help here?


Dan

--
Fedora and Red Hat package maintainer

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 09:07 AM
Oliver Falk
 
Default caching packages on koji builder

Dan Hork wrote:

Hello,

is it possible to somehow cache the packages that are going from koji
hub to the builder to create a build root? There is no NFS connection
between the builder and hub in s390's koji and so each build requires to
download 100 MB via XMLRPC(?)


Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS.

> and this is slow. Will squid help here?

Maybe...

But with a GiB-Connection download of 100MB shouldn't be a problem.
Resolving dependencies, unpacking/writing files from packages to the
buildroot should take longer, than the download....


-of

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 09:36 AM
Dan Horák
 
Default caching packages on koji builder

Oliver Falk p*še v St 05. 11. 2008 v 11:07 +0100:
> Dan Horák wrote:
> > Hello,
> >
> > is it possible to somehow cache the packages that are going from koji
> > hub to the builder to create a build root? There is no NFS connection
> > between the builder and hub in s390's koji and so each build requires to
> > download 100 MB via XMLRPC(?)
>
> Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS.
>
> > and this is slow. Will squid help here?
>
> Maybe...
>
> But with a GiB-Connection download of 100MB shouldn't be a problem.
> Resolving dependencies, unpacking/writing files from packages to the
> buildroot should take longer, than the download....

But the hub and the builders are all at separate locations, so they are
only connected via Internet and the lines are a bit slower :-)


Dan


--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 10:06 AM
Oliver Falk
 
Default caching packages on koji builder

Dan Horák wrote:

Oliver Falk p*še v St 05. 11. 2008 v 11:07 +0100:

Dan Horák wrote:

Hello,

is it possible to somehow cache the packages that are going from koji
hub to the builder to create a build root? There is no NFS connection
between the builder and hub in s390's koji and so each build requires to
download 100 MB via XMLRPC(?)

Newest kojis do not download via XMLRPC. Old kojis 'downloaded' via NFS.

> and this is slow. Will squid help here?

Maybe...

But with a GiB-Connection download of 100MB shouldn't be a problem.
Resolving dependencies, unpacking/writing files from packages to the
buildroot should take longer, than the download....


But the hub and the builders are all at separate locations, so they are
only connected via Internet and the lines are a bit slower :-)


Oh, that's bad. :-( Dunno if you can tell kojid/mock to use a proxy...

-of

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 02:43 PM
Mike Bonnet
 
Default caching packages on koji builder

On Wed, 2008-11-05 at 10:55 +0100, Dan Hork wrote:
> Hello,
>
> is it possible to somehow cache the packages that are going from koji
> hub to the builder to create a build root? There is no NFS connection
> between the builder and hub in s390's koji and so each build requires to
> download 100 MB via XMLRPC(?) and this is slow. Will squid help here?

Koji builders have never downloaded packages via XMLRPC. All
downloading is done by mock/yum, via http (previously nfs).

You could potentially use squid locally to cache downloaded packages.
You'd configure pkgurl in koji.conf to point to your local squid
instance at "http://localhost:8080/koji/packages" or something similar,
and configure squid to pull from the actual http location
where /mnt/koji/packages is being served.


--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 02:49 PM
Jason L Tibbitts III
 
Default caching packages on koji builder

>>>>> "MB" == Mike Bonnet <mikeb@redhat.com> writes:

MB> Koji builders have never downloaded packages via XMLRPC. All
MB> downloading is done by mock/yum, via http (previously nfs).

Well, mock can cache all sorts of things these days. If there are
multiple builders at one location then having a single squid cache for
them all might be nice, but mock's caching would still help to avoid
having to hit the network.

- J<

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 04:03 PM
Mike Bonnet
 
Default caching packages on koji builder

On Wed, 2008-11-05 at 09:49 -0600, Jason L Tibbitts III wrote:
> >>>>> "MB" == Mike Bonnet <mikeb@redhat.com> writes:
>
> MB> Koji builders have never downloaded packages via XMLRPC. All
> MB> downloading is done by mock/yum, via http (previously nfs).
>
> Well, mock can cache all sorts of things these days. If there are
> multiple builders at one location then having a single squid cache for
> them all might be nice, but mock's caching would still help to avoid
> having to hit the network.

Actually mock's caching doesn't really help us. It's all done
per-buildroot, and since every build is run in a different buildroot,
the caches would never be reused. For this reason Koji disables caching
in the mock configs it writes out.

A global (per-machine) rpm cache might be useful for reducing network
bandwidth. However, because mock/yum would have to lock this global
cache while interacting with it, it would become a bottleneck when
running concurrent builds. The best approach currently is probably a
local squid cache.


--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 05:47 PM
Mike McLean
 
Default caching packages on koji builder

Mike Bonnet wrote:

On Wed, 2008-11-05 at 10:55 +0100, Dan Hork wrote:

Hello,

is it possible to somehow cache the packages that are going from koji
hub to the builder to create a build root? There is no NFS connection
between the builder and hub in s390's koji and so each build requires to
download 100 MB via XMLRPC(?) and this is slow. Will squid help here?


Koji builders have never downloaded packages via XMLRPC. All
downloading is done by mock/yum, via http (previously nfs).


This behavior is controlled by kojid options. If you specify the
'topurl' option for kojid, then the mock configs it generates will use
an http:// url to point to the repo. Otherwise it will use a file:// url
(using the value of the 'topdir' option, which defaults to /mnt/koji).


Also, the use of a file:// url doesn't have to mean nfs. You could
theoretically use another shared file system.



You could potentially use squid locally to cache downloaded packages.
You'd configure pkgurl in koji.conf to point to your local squid
instance at "http://localhost:8080/koji/packages" or something similar,
and configure squid to pull from the actual http location
where /mnt/koji/packages is being served.


This is the approach I would recommend.

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 06:19 PM
Dan Horák
 
Default caching packages on koji builder

Mike McLean p*še v St 05. 11. 2008 v 13:47 -0500:
> Mike Bonnet wrote:
> > On Wed, 2008-11-05 at 10:55 +0100, Dan Horák wrote:
> >> Hello,
> >>
> >> is it possible to somehow cache the packages that are going from koji
> >> hub to the builder to create a build root? There is no NFS connection
> >> between the builder and hub in s390's koji and so each build requires to
> >> download 100 MB via XMLRPC(?) and this is slow. Will squid help here?
> >
> > Koji builders have never downloaded packages via XMLRPC. All
> > downloading is done by mock/yum, via http (previously nfs).
>
> This behavior is controlled by kojid options. If you specify the
> 'topurl' option for kojid, then the mock configs it generates will use
> an http:// url to point to the repo. Otherwise it will use a file:// url
> (using the value of the 'topdir' option, which defaults to /mnt/koji).
>
> Also, the use of a file:// url doesn't have to mean nfs. You could
> theoretically use another shared file system.
>
> > You could potentially use squid locally to cache downloaded packages.
> > You'd configure pkgurl in koji.conf to point to your local squid
> > instance at "http://localhost:8080/koji/packages" or something similar,
> > and configure squid to pull from the actual http location
> > where /mnt/koji/packages is being served.
>
> This is the approach I would recommend.

Thanks for all your opinions. I will use the url_rewrite* feature of
squid.


Dan


--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 11-05-2008, 07:14 PM
Mike McLean
 
Default caching packages on koji builder

Mike McLean wrote:
This behavior is controlled by kojid options. If you specify the
'topurl' option for kojid, then the mock configs it generates will use
an http:// url to point to the repo. Otherwise it will use a file:// url
(using the value of the 'topdir' option, which defaults to /mnt/koji).


Also, the use of a file:// url doesn't have to mean nfs. You could
theoretically use another shared file system.


So this is true, but misleading. The interaction of the topurl and
pkgurl options in kojid is complicated.


The topurl/topdir options determine how kojid will locate the repo.
However, with the current code, the repodata will contain url references
for the component rpms. That url is determined when the repo is
generated. This happens during a createrepo task on a builder, and the
pkgurl (not topurl) option is used.


So..
- repodata location determined by topurl/topdir options
- rpm location determined by pkgurl option on the builder that created
the repo.


I admit, this is a bit of a mess.

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 

Thread Tools




All times are GMT. The time now is 12:40 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org