Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Infrastructure (http://www.linux-archive.org/fedora-infrastructure/)
-   -   speeding up the group/dump query in FAS (http://www.linux-archive.org/fedora-infrastructure/423564-speeding-up-group-dump-query-fas.html)

09-06-2010 09:52 PM

speeding up the group/dump query in FAS
 
Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client.
https://admin.fedoraproject.org/accounts/group/dump/

Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc).

Thanks,
Matt

--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Mike McGrath 09-07-2010 02:53 AM

speeding up the group/dump query in FAS
 
On Mon, 6 Sep 2010, Matt_Domsch@Dell.com wrote:

> Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client.
> https://admin.fedoraproject.org/accounts/group/dump/
>
> Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc).
>

This must be related to the alchemy alterations we did recently. I'll
take a look and see what it would take. I was under the assumption this
particular dump was memcached so it should be very quick but I could be
wrong on that.

-Mike
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Toshio Kuratomi 09-07-2010 03:07 PM

speeding up the group/dump query in FAS
 
On Mon, Sep 06, 2010 at 09:53:55PM -0500, Mike McGrath wrote:
> On Mon, 6 Sep 2010, Matt_Domsch@Dell.com wrote:
>
> > Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client.
> > https://admin.fedoraproject.org/accounts/group/dump/
> >
> > Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc).
> >
>
> This must be related to the alchemy alterations we did recently. I'll
> take a look and see what it would take. I was under the assumption this
> particular dump was memcached so it should be very quick but I could be
> wrong on that.
>
I took a look and we should be able to do something similar to what we did
for /groups/list => switch from using code that targets the ORM layer to
code that targets the SQL layer and do the privacy_filtering type of stuff
manually. The tough part is that we probably should figure out how to make
that type of code more generic so that /gorups/list and groups/dump and
anything else that needs it can use the same thing.

It took roughly a day to create the changes to /groups/list -- since we
could copy and paste if we don't figure out how to genericify, probably less
time to get this up and running.

-Toshio
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Ricky Zhou 09-07-2010 03:42 PM

speeding up the group/dump query in FAS
 
On 2010-09-06 04:52:23 PM, Matt_Domsch@Dell.com wrote:
> Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client.
> https://admin.fedoraproject.org/accounts/group/dump/
>
> Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc).
I know that Mike and Toshio are working on a fix to /group/dump, but I
just wanted to mention our JSON interface, which currently doesn't
suffer from the same problem. Would this code work for you?

--
from fedora.client.fas2 import AccountSystem
...
fas = AccountSystem(base_url=BASE_URL, username=USERNAME, password=PASSWORD)
constraints = { 'group': 'packager', 'role_status': 'approved' }
columns = ['username', 'email']

results = fas.people_query(constraints=constraints, columns=columns)
emails = dict((row['username'], row['email']) for row in results)
--

Thanks,
Ricky
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure


All times are GMT. The time now is 07:29 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.