FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Directory

 
 
LinkBack Thread Tools
 
Old 02-03-2011, 03:42 PM
David Boreham
 
Default Performance tuning - where to begin?

On 2/3/2011 9:29 AM, Daniel Fenert wrote:
> There are plenty of configuration options, where should I look first?
>
My first guess would be that you've saturated one core. The server
probably still processes new connections in a single thread (it did on
Unix platforms last time I looked at the code at any rate). So although
the CPU load overall is not 100%, once you max out one core, it won't be
able to process new connections any faster. That said, the rate you are
achieving seems quite low for a modern machine. It would be worthwhile
looking to see if it is doing a reverse DNS lookup on the client IP address.
After that, try using pstack to see what the accept thread is spending
its time on.



--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-03-2011, 03:42 PM
Gerrard Geldenhuis
 
Default Performance tuning - where to begin?

Hi Daniel,
I am getting 1200 conn/sec on very old hardware so maybe something else is wrong.

The very first thing to do is to run logconv.pl script which will come installed with 389. It has a flag for recommendations which I suggest you enable or just enable every flag.

Sample command:
logconv.pl /var/log/dirsrv/slapd-<hostname>/access -efcibaltnxrgjyp

I don't yet understand all the errors that this tool reports but it is a good start.

One other thing, I disabled log buffering at one stage to debug something. I forgot about this and then ended up debugging why the server was so slow until I realized that is what I had done. So make sure you have logbuffering enabled.

dn: cn=config
changetype: modify
replace: nsslapd-accesslog-logbuffering
nsslapd-accesslog-logbuffering: on

Regards

> -----Original Message-----
> From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-
> bounces@lists.fedoraproject.org] On Behalf Of Daniel Fenert
> Sent: 03 February 2011 16:30
> To: General discussion list for the 389 Directory server project.
> Subject: [389-users] Performance tuning - where to begin?
>
> Hi,
>
> I have performance problem on 389-ds server and don't really know where
> to start fine tuning.
>
> My current setup is master (2xQuadCore, 8GB RAM), few read-only slaves.
> It works (more or less) without problems, but I would like to migrate to
> multi master (2 master servers).
>
> To check if one master will handle the whole load, I've tried switching
> clients from slaves to master one by one.
>
> After switching clients from third slave, I've encountered weird problem
> - master was about 50% busy (looking at the cpu, no IO waits), but there was
> problem with new connections.
> Looking at the network level - there was SYN from client, but no ACK until
> one or two retransmissions of SYN.
>
> I've tried increasing thread number (from 30 to 60), but problem still exists.
>
> The problem is near 400-500 connections/second. My whole load is
> ~750conn/sec. Looking at the CPU usage, this server should handle the load.
> It works stable with load ~300conn/sec.
>
> There are plenty of configuration options, where should I look first?
>
> --
> Daniel Fenert
> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users

__________________________________________________ ______________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.

__________________________________________________ ______________________
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-04-2011, 07:29 PM
Andrew Kerr
 
Default Performance tuning - where to begin?

Since the machine isn't even opening up a socket, it makes me think you are hitting either a connection limit or related file limit.

Have you set things such as fs.file-max, tcp keepalive, and file limits (check ulimit) per the docs?

-----Original Message-----
From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of David Boreham
Sent: Thursday, February 03, 2011 11:42 AM
To: 389-users@lists.fedoraproject.org
Subject: Re: [389-users] Performance tuning - where to begin?

On 2/3/2011 9:29 AM, Daniel Fenert wrote:
> There are plenty of configuration options, where should I look first?
>
My first guess would be that you've saturated one core. The server
probably still processes new connections in a single thread (it did on
Unix platforms last time I looked at the code at any rate). So although
the CPU load overall is not 100%, once you max out one core, it won't be
able to process new connections any faster. That said, the rate you are
achieving seems quite low for a modern machine. It would be worthwhile
looking to see if it is doing a reverse DNS lookup on the client IP address.
After that, try using pstack to see what the accept thread is spending
its time on.



--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-04-2011, 07:46 PM
David Boreham
 
Default Performance tuning - where to begin?

On 2/4/2011 1:29 PM, Andrew Kerr wrote:
> Since the machine isn't even opening up a socket, it makes me think you are hitting either a connection limit or related file limit.
It won't open the connection once the listen queue has filled up (if it
did then all that would happen is the box would run out of kernel
memory). The problem is likely that the thread which calls accept() is
not processing the new connections as fast as they arrive. Once that
state is achieved, the queues fill and SYNs will be dropped.


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-04-2011, 08:18 PM
Daniel Fenert
 
Default Performance tuning - where to begin?

W dniu 04.02.2011 21:29, Andrew Kerr pisze:
> Since the machine isn't even opening up a socket, it makes me think you are hitting either a connection limit or related file limit.
>
> Have you set things such as fs.file-max, tcp keepalive, and file limits (check ulimit) per the docs?

Yes, I've set ulimit and fd to 4096. max open fd's about 1500.

> -----Original Message-----
> From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-bounces@lists.fedoraproject.org] On Behalf Of David Boreham
> Sent: Thursday, February 03, 2011 11:42 AM
> To: 389-users@lists.fedoraproject.org
> Subject: Re: [389-users] Performance tuning - where to begin?
>
> On 2/3/2011 9:29 AM, Daniel Fenert wrote:
>> There are plenty of configuration options, where should I look first?
>>
> My first guess would be that you've saturated one core. The server
> probably still processes new connections in a single thread (it did on
> Unix platforms last time I looked at the code at any rate). So although
> the CPU load overall is not 100%, once you max out one core, it won't be
> able to process new connections any faster. That said, the rate you are
> achieving seems quite low for a modern machine. It would be worthwhile
> looking to see if it is doing a reverse DNS lookup on the client IP address.
> After that, try using pstack to see what the accept thread is spending
> its time on.
>
>
>
> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
> you may review at http://www.amdocs.com/email_disclaimer.asp
> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users


--
Daniel Fenert

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-04-2011, 09:01 PM
Daniel Fenert
 
Default Performance tuning - where to begin?

W dniu 04.02.2011 21:46, David Boreham pisze:
> On 2/4/2011 1:29 PM, Andrew Kerr wrote:
>> Since the machine isn't even opening up a socket, it makes me think you are hitting either a connection limit or related file limit.
> It won't open the connection once the listen queue has filled up (if it
> did then all that would happen is the box would run out of kernel
> memory). The problem is likely that the thread which calls accept() is
> not processing the new connections as fast as they arrive. Once that
> state is achieved, the queues fill and SYNs will be dropped.

What else can I do (other than lowering no. of conn/sec)?
I thought 800-1000 conn/s shouldn't be too difficult to achieve on such
hardware.

> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users
--
Daniel Fenert

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 02-10-2011, 10:18 AM
Daniel Fenert
 
Default Performance tuning - where to begin?

W dniu 03.02.2011 17:42, Gerrard Geldenhuis pisze:
> Hi Daniel,
> I am getting 1200 conn/sec on very old hardware so maybe something else is wrong.
>
> The very first thing to do is to run logconv.pl script which will come installed with 389. It has a flag for recommendations which I suggest you enable or just enable every flag.
>
> Sample command:
> logconv.pl /var/log/dirsrv/slapd-<hostname>/access -efcibaltnxrgjyp
>
> I don't yet understand all the errors that this tool reports but it is a good start.
>
> One other thing, I disabled log buffering at one stage to debug something. I forgot about this and then ended up debugging why the server was so slow until I realized that is what I had done. So make sure you have logbuffering enabled.
>
> dn: cn=config
> changetype: modify
> replace: nsslapd-accesslog-logbuffering
> nsslapd-accesslog-logbuffering: on

I've buffered logs, there's no problem with IO (below).

Are you sure that you have 1200 tcp connections/s to LDAP server? If
you've not mistaken connections/s with binds/s, then please tell me
what have you done to get this result?

# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy
id wa st
.....
2 0 105388 15396 28320 6051316 0 0 10 573 5685 7408 24
4 72 0 0
2 0 105388 16012 28312 6050872 0 0 1 718 5891 7812 25
4 72 0 0
1 0 105388 16276 28328 6050336 0 0 8 662 5773 7518 24
4 72 0 0
4 0 105388 16464 28308 6050100 0 0 2 546 5577 7392 24
4 72 0 0
4 0 105388 15424 28324 6051228 0 0 6 665 5724 7480 23
4 73 0 0


> Regards
>
>> -----Original Message-----
>> From: 389-users-bounces@lists.fedoraproject.org [mailto:389-users-
>> bounces@lists.fedoraproject.org] On Behalf Of Daniel Fenert
>> Sent: 03 February 2011 16:30
>> To: General discussion list for the 389 Directory server project.
>> Subject: [389-users] Performance tuning - where to begin?
>>
>> Hi,
>>
>> I have performance problem on 389-ds server and don't really know where
>> to start fine tuning.
>>
>> My current setup is master (2xQuadCore, 8GB RAM), few read-only slaves.
>> It works (more or less) without problems, but I would like to migrate to
>> multi master (2 master servers).
>>
>> To check if one master will handle the whole load, I've tried switching
>> clients from slaves to master one by one.
>>
>> After switching clients from third slave, I've encountered weird problem
>> - master was about 50% busy (looking at the cpu, no IO waits), but there was
>> problem with new connections.
>> Looking at the network level - there was SYN from client, but no ACK until
>> one or two retransmissions of SYN.
>>
>> I've tried increasing thread number (from 30 to 60), but problem still exists.
>>
>> The problem is near 400-500 connections/second. My whole load is
>> ~750conn/sec. Looking at the CPU usage, this server should handle the load.
>> It works stable with load ~300conn/sec.
>>
>> There are plenty of configuration options, where should I look first?
>>
>> --
>> Daniel Fenert
>> --
>> 389 users mailing list
>> 389-users@lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/389-users
> __________________________________________________ ______________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> __________________________________________________ ______________________
> --
> 389 users mailing list
> 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 

Thread Tools




All times are GMT. The time now is 06:54 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org