FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Directory

 
 
LinkBack Thread Tools
 
Old 04-18-2012, 06:10 PM
Russell Beall
 
Default 389 vs Sun DS ldapmodify performance

Does anybody have a pointer to any performance comparisons between Sun DS and 389?
I was extremely happy with the performance boost using 389 on a Linux VM which is 5-8 times faster for ldapsearch operations than the older Sun machines with Sun DS 6.3.
In testing one of our most important use cases just now, I find that the ldapmodify speed is many many times slower. ¬*This doesn't make much sense, so I think I'm doing something wrong or having something misconfigured.
Earlier I improved the write performance by using large db cache sizes and moving the nsslapd-db-home-directory to tmpfs. ¬*Now most modify operations have very little I/O wait except when occasionally flushing the index files and such, and yet, there is a CPU pegged for very long periods of time, orders of magnitude higher than on Sun DS.
Is there any documentation on ldapmodify performance that I could review? ¬*Google searching seems eerily silent on the issue‚Ķ ¬*(which also leads me to believe I have something misconfigured if nobody has been asking about the issue‚Ķ)
The particular use case I am working with involves replacing large quantities of uniqueMember values on entries in ou=groups.
Thanks,Russ.
==============================Russell Beall
Programmer Analyst IVEnterprise Identity ManagementUniversity¬*of¬*Southern Californiabeall@usc.edu=========================== ===



--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-18-2012, 06:15 PM
Rich Megginson
 
Default 389 vs Sun DS ldapmodify performance

On 04/18/2012 12:10 PM, Russell Beall wrote:
Does anybody have a pointer to any performance
comparisons between Sun DS and 389?



I was extremely happy with the performance boost using 389 on
a Linux VM which is 5-8 times faster for ldapsearch operations
than the older Sun machines with Sun DS 6.3.



Please note that earlier versions of the SunDS EULA forbade the
publishing of competitive benchmark figures.







In testing one of our most important use cases just now, I
find that the ldapmodify speed is many many times slower. ¬*This
doesn't make much sense, so I think I'm doing something wrong or
having something misconfigured.



Earlier



You mean, with SunDS?¬* On Linux or Solaris?




I improved the write performance by using large db cache
sizes and moving the nsslapd-db-home-directory to tmpfs. ¬*Now
most modify operations have very little I/O wait except when
occasionally flushing the index files and such, and yet, there
is a CPU pegged for very long periods of time, orders of
magnitude higher than on Sun DS.



Is there any documentation on ldapmodify performance that I
could review? ¬*Google searching seems eerily silent on the
issue‚Ķ ¬*(which also leads me to believe I have something
misconfigured if nobody has been asking about the issue…)



The particular use case I am working with involves replacing
large quantities of uniqueMember values on entries in ou=groups.



Yeah, this particular operation has not been optimized.¬* I believe
SunDS added explicit optimizations for this particular case.







Thanks,
Russ.



==============================




Russell Beall

Programmer Analyst IV
Enterprise Identity
Management
University¬*of¬*Southern
California
beall@usc.edu



==============================


















--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users





--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-18-2012, 07:33 PM
Michael Gettes
 
Default 389 vs Sun DS ldapmodify performance

Hey russ, I've got the same problem for large groups using member... We are coming from an openldap world so not much use of uniquemember yet.

On Apr 18, 2012 2:10 PM, "Russell Beall" <beall@usc.edu> wrote:
Does anybody have a pointer to any performance comparisons between Sun DS and 389?
I was extremely happy with the performance boost using 389 on a Linux VM which is 5-8 times faster for ldapsearch operations than the older Sun machines with Sun DS 6.3.

In testing one of our most important use cases just now, I find that the ldapmodify speed is many many times slower. ¬*This doesn't make much sense, so I think I'm doing something wrong or having something misconfigured.

Earlier I improved the write performance by using large db cache sizes and moving the nsslapd-db-home-directory to tmpfs. ¬*Now most modify operations have very little I/O wait except when occasionally flushing the index files and such, and yet, there is a CPU pegged for very long periods of time, orders of magnitude higher than on Sun DS.

Is there any documentation on ldapmodify performance that I could review? ¬*Google searching seems eerily silent on the issue‚Ķ ¬*(which also leads me to believe I have something misconfigured if nobody has been asking about the issue‚Ķ)

The particular use case I am working with involves replacing large quantities of uniqueMember values on entries in ou=groups.
Thanks,Russ.
==============================


Russell Beall
Programmer Analyst IVEnterprise Identity ManagementUniversity¬*of¬*Southern California
beall@usc.edu==============================









--

389 users mailing list

389-users@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/389-users

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-18-2012, 07:40 PM
Rich Megginson
 
Default 389 vs Sun DS ldapmodify performance

On 04/18/2012 01:33 PM, Michael Gettes wrote:


Hey russ, I've got the same problem for large groups using
member... We are coming from an openldap world so not much use
of uniquemember yet.




It's essentially the same problem - it doesn't matter if you use
member or uniquemember.




On Apr 18, 2012 2:10 PM, "Russell Beall"
<beall@usc.edu>
wrote:

Does anybody have a pointer
to any performance comparisons between Sun DS and 389?



I was extremely happy with the performance boost using
389 on a Linux VM which is 5-8 times faster for ldapsearch
operations than the older Sun machines with Sun DS 6.3.



In testing one of our most important use cases just
now, I find that the ldapmodify speed is many many times
slower. ¬*This doesn't make much sense, so I think I'm
doing something wrong or having something misconfigured.



Earlier I improved the write performance by using large
db cache sizes and moving the nsslapd-db-home-directory to
tmpfs. ¬*Now most modify operations have very little I/O
wait except when occasionally flushing the index files and
such, and yet, there is a CPU pegged for very long periods
of time, orders of magnitude higher than on Sun DS.



Is there any documentation on ldapmodify performance
that I could review? ¬*Google searching seems eerily silent
on the issue‚Ķ ¬*(which also leads me to believe I have
something misconfigured if nobody has been asking about
the issue…)



The particular use case I am working with involves
replacing large quantities of uniqueMember values on
entries in ou=groups.



Thanks,
Russ.



==============================





Russell Beall

Programmer Analyst IV
Enterprise
Identity Management
University¬*of¬*Southern
California

beall@usc.edu



==============================




















--

389 users mailing list

389-users@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/389-users








--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users





--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-18-2012, 09:06 PM
Russell Beall
 
Default 389 vs Sun DS ldapmodify performance

On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote:You mean, with SunDS?* On Linux or Solaris?

The Sun DS instance I'm talking about is on Sun v440 machines.
The 389 instance is on relatively new servers with RedHat Linux 6.
Yeah, this particular operation has not been optimized.* I believe SunDS added explicit optimizations for this particular case.

It is becoming painfully apparent as I write more detailed tests. *389 takes time to add or delete uniquemember values proportionate to the number of values being operated on and is using about twice as much time to delete as it does to add. *Sun DS appears to have perhaps an almost O(1) algorithm in play on both adding and deleting values.
Is there perhaps some kind of referential integrity setting that is being used and forcing some kind of lookup of each value, one that we could perhaps turn off? *We wouldn't need such a check because our metadirectory process handles the integrity/consistency checking already.
This behavior is especially surprising given that I can run ldapdelete on the entry followed by ldapadd and the changes are almost instantaneous. *I can delete and then re-add all 852 groups, some with 80K+ unique member attributes, in about ten minutes (using 389).
Why would membership changes on the group take forever to process if the entire group can be reconstructed in an instant?
Could this operation be optimized internally by turning the ldapmodify into a two-step process of ldapdelete followed by ldapadd (locking the entry during the processing and not actually replicating it that way so the group never appears to be missing)?
Thanks,Russ.
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-19-2012, 09:20 AM
Andrey Ivanov
 
Default 389 vs Sun DS ldapmodify performance

Hi Russel,


Le 18 avril 2012 23:06, Russell Beall <beall@usc.edu> a ťcrit :


On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote:Yeah, this particular operation has not been optimized.* I believe SunDS added explicit optimizations for this particular case.






It is becoming painfully apparent as I write more detailed tests. *389 takes time to add or delete uniquemember values proportionate to the number of values being operated on and is using about twice as much time to delete as it does to add. *Sun DS appears to have perhaps an almost O(1) algorithm in play on both adding and deleting values.




Is there perhaps some kind of referential integrity setting that is being used and forcing some kind of lookup of each value, one that we could perhaps turn off? *We wouldn't need such a check because our metadirectory process handles the integrity/consistency checking already.


There is memberOf plugin that maintains the memberOf attribute for groups. I don't know whether* it is activated by default or not. You could try to disable it. There is also referential integrity plugin, attribute uniqueness plugin, maybe USN plugin or custom indexes that could consume a lot of CPU. Make sure you've disabled them if you don't need them.



@+

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-19-2012, 11:47 AM
"Andrea Modesto Rossi"
 
Default 389 vs Sun DS ldapmodify performance

Dear all, i've got a little problem using the new wdmin Console (
http://port389.org/download/389-Console-1.1.6-i386.msi ) with my 389
Server:

[root@pserver ~]# rpm -qa | grep 389
389-ds-base-1.2.9.9-1.el5
389-ds-console-1.2.6-1.el5
389-dsgw-1.1.7-2.el5
389-console-1.1.7-3.el5
389-ds-console-doc-1.2.6-1.el5
389-admin-1.1.23-1.el5
389-admin-console-1.1.8-1.el5
389-admin-console-doc-1.1.8-1.el5
389-ds-1.2.1-1.el5
389-ds-base-libs-1.2.9.9-1.el5
389-adminutil-1.1.14-1.el5

When i try to connect to Directory Server, client issues an error called
"Class Loader Error" with this txt:

" Failed to install a local copy of 389-ds-123.jar or one of its
supporting files. ...."


What happen? Can anyone help me? Anyway i didn't have this error with the
previous version of Admin Console (i think 1.1.4).... i'm looking in the
website where is that file, but there isn't.

Thanks a lot.

Have a nice day,





--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-19-2012, 04:50 PM
Russell Beall
 
Default 389 vs Sun DS ldapmodify performance

Thanks for the tips. *I scanned the dse.ldif for those plugins and I found definitions for them all, but they all have nsslapd-pluginEnabled: off.
There is something special about the uniquemember attribute that requires additional processing different from other attributes... *Ldapmodify of other attributes runs pretty quick.
Regards,Russ.
On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote:Hi Russel,


Le 18 avril 2012 23:06, Russell Beall <beall@usc.edu> a ťcrit :


On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote:Yeah, this particular operation has not been optimized.* I believe SunDS added explicit optimizations for this particular case.






It is becoming painfully apparent as I write more detailed tests. *389 takes time to add or delete uniquemember values proportionate to the number of values being operated on and is using about twice as much time to delete as it does to add. *Sun DS appears to have perhaps an almost O(1) algorithm in play on both adding and deleting values.




Is there perhaps some kind of referential integrity setting that is being used and forcing some kind of lookup of each value, one that we could perhaps turn off? *We wouldn't need such a check because our metadirectory process handles the integrity/consistency checking already.


There is memberOf plugin that maintains the memberOf attribute for groups. I don't know whether* it is activated by default or not. You could try to disable it. There is also referential integrity plugin, attribute uniqueness plugin, maybe USN plugin or custom indexes that could consume a lot of CPU. Make sure you've disabled them if you don't need them.



@+

--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-19-2012, 05:18 PM
Rich Megginson
 
Default 389 vs Sun DS ldapmodify performance

On 04/19/2012 10:50 AM, Russell Beall wrote:
Thanks for the tips. *I scanned the dse.ldif for those
plugins and I found definitions for them all, but they all have
nsslapd-pluginEnabled: off.



There is something special about the uniquemember attribute
that requires additional processing different from other
attributes... *Ldapmodify of other attributes runs pretty quick.



Is uniquemember the only attribute using large numbers of multiple
values in ldapmodify operations?







Regards,
Russ.



On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote:

Hi Russel,





Le 18 avril 2012 23:06, Russell
Beall <beall@usc.edu>
a ťcrit :




On Apr 18, 2012, at 11:15 AM, Rich Megginson
wrote:

Yeah,
this particular operation has not been
optimized.* I believe SunDS added explicit
optimizations for this particular case.











It
is becoming painfully apparent as I write more
detailed tests. *389 takes time to add or delete
uniquemember values proportionate to the number of
values being operated on and is using about twice
as much time to delete as it does to add. *Sun DS
appears to have perhaps an almost O(1) algorithm
in play on both adding and deleting values.



Is
there perhaps some kind of referential integrity
setting that is being used and forcing some kind
of lookup of each value, one that we could perhaps
turn off? *We wouldn't need such a check because
our metadirectory process handles the
integrity/consistency checking already.




There is memberOf plugin that maintains the memberOf
attribute for groups. I don't know whether* it is
activated by default or not. You could try to disable
it. There is also referential integrity plugin,
attribute uniqueness plugin, maybe USN plugin or custom
indexes that could consume a lot of CPU. Make sure
you've disabled them if you don't need them.



@+



--

389 users mailing list

389-users@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/389-users









--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users





--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 04-19-2012, 05:38 PM
Russell Beall
 
Default 389 vs Sun DS ldapmodify performance

That is by far the largest example we have. *We use groups with the uniquemember attribute linking to account entries and more than a few of the groups have tens of thousands of values for uniquemember. *We create more of these groups regularly and it will be a problem for it to take many hours to construct such a group versus seconds/minutes with Sun DS. *Our metadirectory process does not use ldapadd to create the group pre-populated, the group is created and then ldapmodify is run to add members. *Three times a year at the change of semester, many thousands of group membership changes are processed and we already have a problem with it taking multiple days to process the entire set...
We also have large quantities of eduPersonEntitlement on account records, but those sets are not nearly as numerously populated. *I can delete and re-add the eduPersonEntitlement attribute across 110,000 records in about 40 minutes (20 minutes each way -- with 389).

Russ.
On Apr 19, 2012, at 10:18 AM, Rich Megginson wrote:




On 04/19/2012 10:50 AM, Russell Beall wrote:
Thanks for the tips. *I scanned the dse.ldif for those
plugins and I found definitions for them all, but they all have
nsslapd-pluginEnabled: off.



There is something special about the uniquemember attribute
that requires additional processing different from other
attributes... *Ldapmodify of other attributes runs pretty quick.



Is uniquemember the only attribute using large numbers of multiple
values in ldapmodify operations?







Regards,
Russ.



On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote:

Hi Russel,





Le 18 avril 2012 23:06, Russell
Beall <beall@usc.edu>
a ťcrit :




On Apr 18, 2012, at 11:15 AM, Rich Megginson
wrote:

Yeah,
this particular operation has not been
optimized.* I believe SunDS added explicit
optimizations for this particular case.











It
is becoming painfully apparent as I write more
detailed tests. *389 takes time to add or delete
uniquemember values proportionate to the number of
values being operated on and is using about twice
as much time to delete as it does to add. *Sun DS
appears to have perhaps an almost O(1) algorithm
in play on both adding and deleting values.



Is
there perhaps some kind of referential integrity
setting that is being used and forcing some kind
of lookup of each value, one that we could perhaps
turn off? *We wouldn't need such a check because
our metadirectory process handles the
integrity/consistency checking already.




There is memberOf plugin that maintains the memberOf
attribute for groups. I don't know whether* it is
activated by default or not. You could try to disable
it. There is also referential integrity plugin,
attribute uniqueness plugin, maybe USN plugin or custom
indexes that could consume a lot of CPU. Make sure
you've disabled them if you don't need them.



@+



--

389 users mailing list

389-users@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/389-users









--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users






--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 

Thread Tools




All times are GMT. The time now is 11:28 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org