FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 07-23-2008, 07:56 PM
Naveen Gavini
 
Default package upload/import speed

Hello all,

When importing or building RPM's from source is there an option that is
throttling the upload speed of the package?
We are seeing speeds of 10kbps which makes importing/building large
packages very slow.


Thanks.

--
Naveen Gavini
Student Systems Programmer
OSS/CSS - OIT Rutgers
ngavini@jla.rutgers.edu

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 07-23-2008, 08:29 PM
Bryce
 
Default package upload/import speed

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Naveen Gavini wrote:
> Hello all,
>
> When importing or building RPM's from source is there an option that is
> throttling the upload speed of the package?
> We are seeing speeds of 10kbps which makes importing/building large
> packages very slow.
>
> Thanks.
>

On a local machine (where client and server, hub etc all reside) I get
140K/s (about x14 what you're seeing) but that seems to be an artificial
cap given that it's invariant no matter what the loading on the machine
is. Ie it's not CPU bound. Unfortunately it's a royal PITA for scratch
building since that means large src rpms like the kernel can take up to
5 minutes just to be 'uploaded' on the same damned box.

To be honest this is something I've been scratching my head over as
well. Various slim picked information I've gathered tries to point the
finger of slowness at SSL itself for the encryption speed and suggest
getting hold of SSL HW accelerators but I don't buy that argument due to
the constant transfer speed.

I haven't been able to decide if it's a client side issue or a server
side issue though 8/

Phil
=--=
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIh5SgPcECCTpcVR0RAjZwAJ9s0giafZQmRHtxdyOefv q+QktiCgCfeFe+
hoOV7CUIucOTbZI1PTtFgDs=
=gMPu
-----END PGP SIGNATURE-----

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 07-23-2008, 08:34 PM
Mike Bonnet
 
Default package upload/import speed

On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
> Hello all,
>
> When importing or building RPM's from source is there an option that is
> throttling the upload speed of the package?
> We are seeing speeds of 10kbps which makes importing/building large
> packages very slow.

Package upload happens via xml-rpc, and to avoid holding the entire
contents of the file in memory, it's done in chunks. Unfortunately
python xml-rpc libs don't support keep-alive, so if you're using SSL
authentication you incur the non-trivial SSL connection setup/teardown
overhead for every chunk, in addition to the TCP connection
setup/teardown. The chunk size in your version is probably very small
(64k?). That has been increased in the Koji repo, but not released yet.
The next version will improve this situation somewhat.


--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 07-23-2008, 09:03 PM
Toshio Kuratomi
 
Default package upload/import speed

Mike Bonnet wrote:

On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:

Hello all,

When importing or building RPM's from source is there an option that is
throttling the upload speed of the package?
We are seeing speeds of 10kbps which makes importing/building large
packages very slow.


Package upload happens via xml-rpc, and to avoid holding the entire
contents of the file in memory, it's done in chunks. Unfortunately
python xml-rpc libs don't support keep-alive, so if you're using SSL
authentication you incur the non-trivial SSL connection setup/teardown
overhead for every chunk, in addition to the TCP connection
setup/teardown. The chunk size in your version is probably very small
(64k?). That has been increased in the Koji repo, but not released yet.
The next version will improve this situation somewhat.

Any chance we could port to pycurl which does support keep-alive? Or is
urllib pretty deeply embedded in the code ATM?


-Toshio

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 07-23-2008, 11:02 PM
Mike Bonnet
 
Default package upload/import speed

On Wed, 2008-07-23 at 14:03 -0700, Toshio Kuratomi wrote:
> Mike Bonnet wrote:
> > On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:
> >> Hello all,
> >>
> >> When importing or building RPM's from source is there an option that is
> >> throttling the upload speed of the package?
> >> We are seeing speeds of 10kbps which makes importing/building large
> >> packages very slow.
> >
> > Package upload happens via xml-rpc, and to avoid holding the entire
> > contents of the file in memory, it's done in chunks. Unfortunately
> > python xml-rpc libs don't support keep-alive, so if you're using SSL
> > authentication you incur the non-trivial SSL connection setup/teardown
> > overhead for every chunk, in addition to the TCP connection
> > setup/teardown. The chunk size in your version is probably very small
> > (64k?). That has been increased in the Koji repo, but not released yet.
> > The next version will improve this situation somewhat.
> >
> Any chance we could port to pycurl which does support keep-alive? Or is
> urllib pretty deeply embedded in the code ATM?

I don't know anything about pycurl, but the transport layer is decently
separated from the XML-RPC layer in xmlrpclib, so it should be possible
to replace that it with a transport that supports SSL and keep-alive. I
haven't looked at how much work this might be though.

> -Toshio
>
> --
> Fedora-buildsys-list mailing list
> Fedora-buildsys-list@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 
Old 07-24-2008, 06:56 AM
Oliver Falk
 
Default package upload/import speed

Mike Bonnet wrote:

On Wed, 2008-07-23 at 15:56 -0400, Naveen Gavini wrote:

Hello all,

When importing or building RPM's from source is there an option that is
throttling the upload speed of the package?
We are seeing speeds of 10kbps which makes importing/building large
packages very slow.


Package upload happens via xml-rpc, and to avoid holding the entire
contents of the file in memory, it's done in chunks. Unfortunately
python xml-rpc libs don't support keep-alive, so if you're using SSL
authentication you incur the non-trivial SSL connection setup/teardown
overhead for every chunk, in addition to the TCP connection
setup/teardown. The chunk size in your version is probably very small
(64k?). That has been increased in the Koji repo, but not released yet.
The next version will improve this situation somewhat.


Just wanted to write the same...

Naveen, you could increase your blocksiize as a workaround. Just search
for blocksize in /usr/bin/koji (function do_import)... I've done that
here and it helps. BUT do not set to too large, as for every chunk some
checksum (it seems) is being build (SSL does that I think) and the
larger your chunk the more CPU resources it will use!!!


-of

--
Fedora-buildsys-list mailing list
Fedora-buildsys-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
 

Thread Tools




All times are GMT. The time now is 12:15 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org