FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Directory

 
 
LinkBack Thread Tools
 
Old 05-16-2012, 08:09 PM
Brad Schuetz
 
Default Strange Disk IO issue

On 05/16/2012 11:54 AM, Nathan Kinder wrote:
> On 05/16/2012 11:19 AM, Brad Schuetz wrote:
>>
>> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
>>> The exact timing of the issue is to strange is there a backup job
>>> running at midnight. Or some other timed job that could be eating the
>>> ram or disk IO. Possibly one that is reliant on ldap queries that
>>> would otherwise be inocuious.
>>>
>>>
>> It doesn't happen at midnight, it's 24 hours from when the process was
>> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
>> around 3:17pm on Thursday that server will go to 100% disk IO usage.
> The default tombstone purge interval is 1 day, which seems to fit what
> you are seeing. The tombstone reap thread will start every 24 hours
> to find tombstone entries that can be deleted. The default retention
> period for tombstones is 1 week. It is possible that you have a large
> number of tombstone entries that need to be deleted. This will occur
> independently on all of your server instances. This is controlled by
> the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"
> attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
> tree,cn=config" entry.
>
I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
that default), and "nsDS5ReplicaPurgeDelay" is set to 3600


> You can search for "(objectclass=nstombstone)" as Directory Manager to
> see how many tombstone entries you have.

I have a LOT of tombstone entries, over 200k on this one server (I'm
guessing since I've been restarting the process for over a week now, not
letting it run the cleanup process).

So, any suggestions on what can I do to fix this? The process that's
reaping the entries is using too much IO making queries time out, older
versions of the software did not exhibit this behavior. In fact, I can
reinitalize the entire replica faster than this thing is reaping the
entries, it takes 7 minutes to reinit a replica, but when this issue
first started I let the dirsrv run much longer before restarting it.

Should I make it purge more frequently so there are fewer entries to
reap? Or is this just some weird bug?

--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-16-2012, 10:06 PM
Nathan Kinder
 
Default Strange Disk IO issue

On 05/16/2012 01:09 PM, Brad Schuetz wrote:

On 05/16/2012 11:54 AM, Nathan Kinder wrote:

On 05/16/2012 11:19 AM, Brad Schuetz wrote:

On 05/16/2012 06:16 AM, Paul Robert Marino wrote:

The exact timing of the issue is to strange is there a backup job
running at midnight. Or some other timed job that could be eating the
ram or disk IO. Possibly one that is reliant on ldap queries that
would otherwise be inocuious.



It doesn't happen at midnight, it's 24 hours from when the process was
started, so I can restart dirsrv at 3:17pm on Wednesday and at right
around 3:17pm on Thursday that server will go to 100% disk IO usage.

The default tombstone purge interval is 1 day, which seems to fit what
you are seeing. The tombstone reap thread will start every 24 hours
to find tombstone entries that can be deleted. The default retention
period for tombstones is 1 week. It is possible that you have a large
number of tombstone entries that need to be deleted. This will occur
independently on all of your server instances. This is controlled by
the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"
attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
tree,cn=config" entry.


I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
Ok, so this means every 24 hours, the tombstone reap thread will look
for tombstones older than 1 hour and remove them.




You can search for "(objectclass=nstombstone)" as Directory Manager to
see how many tombstone entries you have.

I have a LOT of tombstone entries, over 200k on this one server (I'm
guessing since I've been restarting the process for over a week now, not
letting it run the cleanup process).
That's possible if you really do 200k delete operations in 1 week, but
that sounds like a lot. It would seem that these tombstones have been
building up for a longer time than 1 week.


So, any suggestions on what can I do to fix this? The process that's
reaping the entries is using too much IO making queries time out, older
versions of the software did not exhibit this behavior. In fact, I can
reinitalize the entire replica faster than this thing is reaping the
entries, it takes 7 minutes to reinit a replica, but when this issue
first started I let the dirsrv run much longer before restarting it.
Due to the number of matching entries for the tombstone search, it is
having to walk your entire database, which is why you see the IO
spiking. What you could do is to export your database with "db2ldif
-r". This will preserve the replication related data in the LDIF. You
can then remove the tombstone entries in the LDIF file via a script and
reimport it. You would have to do this on each server, or do it on one
master and then reinitialize the rest of your servers. One thing to
watch out for is that you do not want to remove the RUV entry, which
will have the "nstombstone" objectclass. This RUV entry will have a
"nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff" value that you can use
to distinguish if from the rest of the tombstones.


Should I make it purge more frequently so there are fewer entries to
reap? Or is this just some weird bug?

I'd leave the purge settings as they are.


--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-16-2012, 11:01 PM
Rich Megginson
 
Default Strange Disk IO issue

On 05/16/2012 04:06 PM, Nathan Kinder wrote:

On 05/16/2012 01:09 PM, Brad Schuetz wrote:

On 05/16/2012 11:54 AM, Nathan Kinder wrote:

On 05/16/2012 11:19 AM, Brad Schuetz wrote:

On 05/16/2012 06:16 AM, Paul Robert Marino wrote:

The exact timing of the issue is to strange is there a backup job
running at midnight. Or some other timed job that could be eating the
ram or disk IO. Possibly one that is reliant on ldap queries that
would otherwise be inocuious.



It doesn't happen at midnight, it's 24 hours from when the process was
started, so I can restart dirsrv at 3:17pm on Wednesday and at right
around 3:17pm on Thursday that server will go to 100% disk IO usage.

The default tombstone purge interval is 1 day, which seems to fit what
you are seeing. The tombstone reap thread will start every 24 hours
to find tombstone entries that can be deleted. The default retention
period for tombstones is 1 week. It is possible that you have a large
number of tombstone entries that need to be deleted. This will occur
independently on all of your server instances. This is controlled by
the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"
attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
tree,cn=config" entry.


I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
Ok, so this means every 24 hours, the tombstone reap thread will look
for tombstones older than 1 hour and remove them.




You can search for "(objectclass=nstombstone)" as Directory Manager to
see how many tombstone entries you have.

I have a LOT of tombstone entries, over 200k on this one server (I'm
guessing since I've been restarting the process for over a week now, not
letting it run the cleanup process).
That's possible if you really do 200k delete operations in 1 week, but
that sounds like a lot. It would seem that these tombstones have been
building up for a longer time than 1 week.


So, any suggestions on what can I do to fix this? The process that's
reaping the entries is using too much IO making queries time out, older
versions of the software did not exhibit this behavior. In fact, I can
reinitalize the entire replica faster than this thing is reaping the
entries, it takes 7 minutes to reinit a replica, but when this issue
first started I let the dirsrv run much longer before restarting it.
Due to the number of matching entries for the tombstone search, it is
having to walk your entire database, which is why you see the IO spiking.


Perhaps also try increasing nsslapd-idlistscanlimit so that it can hold
the entire candidate list of tombstones to delete -

http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Indexes.html#About_Indexes-Overview_of_the_Searching_Algorithm
What you could do is to export your database with "db2ldif -r". This
will preserve the replication related data in the LDIF. You can then
remove the tombstone entries in the LDIF file via a script and
reimport it. You would have to do this on each server, or do it on
one master and then reinitialize the rest of your servers. One thing
to watch out for is that you do not want to remove the RUV entry,
which will have the "nstombstone" objectclass. This RUV entry will
have a "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff" value that you
can use to distinguish if from the rest of the tombstones.


Should I make it purge more frequently so there are fewer entries to
reap? Or is this just some weird bug?

I'd leave the purge settings as they are.


--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 12:48 AM
Brad Schuetz
 
Default Strange Disk IO issue

On 05/16/2012 04:01 PM, Rich Megginson wrote:
> On 05/16/2012 04:06 PM, Nathan Kinder wrote:
>> On 05/16/2012 01:09 PM, Brad Schuetz wrote:
>>> On 05/16/2012 11:54 AM, Nathan Kinder wrote:
>>>> On 05/16/2012 11:19 AM, Brad Schuetz wrote:
>>>>> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
>>>>>> The exact timing of the issue is to strange is there a backup job
>>>>>> running at midnight. Or some other timed job that could be eating
>>>>>> the
>>>>>> ram or disk IO. Possibly one that is reliant on ldap queries that
>>>>>> would otherwise be inocuious.
>>>>>>
>>>>>>
>>>>> It doesn't happen at midnight, it's 24 hours from when the process
>>>>> was
>>>>> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
>>>>> around 3:17pm on Thursday that server will go to 100% disk IO usage.
>>>> The default tombstone purge interval is 1 day, which seems to fit what
>>>> you are seeing. The tombstone reap thread will start every 24 hours
>>>> to find tombstone entries that can be deleted. The default retention
>>>> period for tombstones is 1 week. It is possible that you have a large
>>>> number of tombstone entries that need to be deleted. This will occur
>>>> independently on all of your server instances. This is controlled by
>>>> the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"
>>>> attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
>>>> tree,cn=config" entry.
>>>>
>>> I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
>>> that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
>> Ok, so this means every 24 hours, the tombstone reap thread will look
>> for tombstones older than 1 hour and remove them.
>>>
>>>
>>>> You can search for "(objectclass=nstombstone)" as Directory Manager to
>>>> see how many tombstone entries you have.
>>> I have a LOT of tombstone entries, over 200k on this one server (I'm
>>> guessing since I've been restarting the process for over a week now,
>>> not
>>> letting it run the cleanup process).
>> That's possible if you really do 200k delete operations in 1 week,
>> but that sounds like a lot. It would seem that these tombstones have
>> been building up for a longer time than 1 week.
>>>
>>> So, any suggestions on what can I do to fix this? The process that's
>>> reaping the entries is using too much IO making queries time out, older
>>> versions of the software did not exhibit this behavior. In fact, I can
>>> reinitalize the entire replica faster than this thing is reaping the
>>> entries, it takes 7 minutes to reinit a replica, but when this issue
>>> first started I let the dirsrv run much longer before restarting it.
>> Due to the number of matching entries for the tombstone search, it is
>> having to walk your entire database, which is why you see the IO
>> spiking.
>
> Perhaps also try increasing nsslapd-idlistscanlimit so that it can
> hold the entire candidate list of tombstones to delete -
> http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Indexes.html#About_Indexes-Overview_of_the_Searching_Algorithm
>

Is there any way that I can remove the nsTombstone entries from the
master server so I can get this under control? I think I found out why
I have so many nsTombstone entries in the first place so I'd like to get
the current ones deleted and see how much delete activity I'll have
moving forward.

I saw this bug report,
<https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
indicate that I should be able to ldapdelete the nsTombstone entries
using their full dn, however it always says object not found.

I'd rather not fully export and reimport the master and then reinit all
the replicas if I can avoid it.

--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 01:24 AM
Rich Megginson
 
Default Strange Disk IO issue

On 05/16/2012 06:48 PM, Brad Schuetz wrote:

On 05/16/2012 04:01 PM, Rich Megginson wrote:

On 05/16/2012 04:06 PM, Nathan Kinder wrote:

On 05/16/2012 01:09 PM, Brad Schuetz wrote:

On 05/16/2012 11:54 AM, Nathan Kinder wrote:

On 05/16/2012 11:19 AM, Brad Schuetz wrote:

On 05/16/2012 06:16 AM, Paul Robert Marino wrote:

The exact timing of the issue is to strange is there a backup job
running at midnight. Or some other timed job that could be eating
the
ram or disk IO. Possibly one that is reliant on ldap queries that
would otherwise be inocuious.



It doesn't happen at midnight, it's 24 hours from when the process
was
started, so I can restart dirsrv at 3:17pm on Wednesday and at right
around 3:17pm on Thursday that server will go to 100% disk IO usage.

The default tombstone purge interval is 1 day, which seems to fit what
you are seeing. The tombstone reap thread will start every 24 hours
to find tombstone entries that can be deleted. The default retention
period for tombstones is 1 week. It is possible that you have a large
number of tombstone entries that need to be deleted. This will occur
independently on all of your server instances. This is controlled by
the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"
attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
tree,cn=config" entry.


I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
that default), and "nsDS5ReplicaPurgeDelay" is set to 3600

Ok, so this means every 24 hours, the tombstone reap thread will look
for tombstones older than 1 hour and remove them.



You can search for "(objectclass=nstombstone)" as Directory Manager to
see how many tombstone entries you have.

I have a LOT of tombstone entries, over 200k on this one server (I'm
guessing since I've been restarting the process for over a week now,
not
letting it run the cleanup process).

That's possible if you really do 200k delete operations in 1 week,
but that sounds like a lot. It would seem that these tombstones have
been building up for a longer time than 1 week.

So, any suggestions on what can I do to fix this? The process that's
reaping the entries is using too much IO making queries time out, older
versions of the software did not exhibit this behavior. In fact, I can
reinitalize the entire replica faster than this thing is reaping the
entries, it takes 7 minutes to reinit a replica, but when this issue
first started I let the dirsrv run much longer before restarting it.

Due to the number of matching entries for the tombstone search, it is
having to walk your entire database, which is why you see the IO
spiking.

Perhaps also try increasing nsslapd-idlistscanlimit so that it can
hold the entire candidate list of tombstones to delete -
http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Managing_Indexes.html#About_Indexes-Overview_of_the_Searching_Algorithm


Is there any way that I can remove the nsTombstone entries from the
master server so I can get this under control? I think I found out why
I have so many nsTombstone entries in the first place so I'd like to get
the current ones deleted and see how much delete activity I'll have
moving forward.

I saw this bug report,
<https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
indicate that I should be able to ldapdelete the nsTombstone entries
using their full dn, however it always says object not found.


Can you provide
1) the exact ldapdelete command line
2) the excerpts from the access and errors log from around the time of
the deletion





I'd rather not fully export and reimport the master and then reinit all
the replicas if I can avoid it.

--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 01:48 AM
Brad Schuetz
 
Default Strange Disk IO issue

On 05/16/2012 06:24 PM, Rich Megginson wrote:
> On 05/16/2012 06:48 PM, Brad Schuetz wrote:
>> Is there any way that I can remove the nsTombstone entries from the
>> master server so I can get this under control? I think I found out why
>> I have so many nsTombstone entries in the first place so I'd like to get
>> the current ones deleted and see how much delete activity I'll have
>> moving forward.
>>
>> I saw this bug report,
>> <https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
>> indicate that I should be able to ldapdelete the nsTombstone entries
>> using their full dn, however it always says object not found.
>
> Can you provide
> 1) the exact ldapdelete command line
> 2) the excerpts from the access and errors log from around the time of
> the deletion

Here's the command with only the username part of the DN redacted and
the access log for the command, I confirmed that after the delete failed
that the entry was in fact still there by running the ldap search again
for nstombstone entries to be sure that it wasn't just a timing issue.
Also attempted the same command with quotes around the DN to delete.
The error log produced no entries around the time of the command being run.

ldapdelete -x -D 'cn=Directory Manager' -W
nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,d c=omnis,dc=com

This was the output from the above command:

ldap_delete: No such object (32)
matched DN:
uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com

access log:
17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
::1 to ::1
[17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
[17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
Manager" method=128 version=3
[17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
nentries=0 etime=0 dn="cn=directory manager"
[17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
[17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1

However I can report that the internal process seems to have finally
cleaned up the bulk of the entries, I had over 20k entries on the master
and it's finally down to less than 100. I temporarily changed the
values for when it should reap the entries to try and get it to clear
them out internally after I fixed the issue that I believe was
responsible for the extraneous deletes.

--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 02:06 AM
Rich Megginson
 
Default Strange Disk IO issue

On 05/16/2012 07:48 PM, Brad Schuetz wrote:

On 05/16/2012 06:24 PM, Rich Megginson wrote:

On 05/16/2012 06:48 PM, Brad Schuetz wrote:

Is there any way that I can remove the nsTombstone entries from the
master server so I can get this under control? I think I found out why
I have so many nsTombstone entries in the first place so I'd like to get
the current ones deleted and see how much delete activity I'll have
moving forward.

I saw this bug report,
<https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
indicate that I should be able to ldapdelete the nsTombstone entries
using their full dn, however it always says object not found.

Can you provide
1) the exact ldapdelete command line
2) the excerpts from the access and errors log from around the time of
the deletion

Here's the command with only the username part of the DN redacted and
the access log for the command, I confirmed that after the delete failed
that the entry was in fact still there by running the ldap search again
for nstombstone entries to be sure that it wasn't just a timing issue.
Also attempted the same command with quotes around the DN to delete.
The error log produced no entries around the time of the command being run.

ldapdelete -x -D 'cn=Directory Manager' -W
nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,d c=omnis,dc=com

This was the output from the above command:

ldap_delete: No such object (32)
matched DN:
uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com

Looks like 617862 is back . . .


access log:
17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
::1 to ::1
[17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
[17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
Manager" method=128 version=3
[17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
nentries=0 etime=0 dn="cn=directory manager"
[17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
[17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1

However I can report that the internal process seems to have finally
cleaned up the bulk of the entries, I had over 20k entries on the master
and it's finally down to less than 100. I temporarily changed the
values for when it should reap the entries to try and get it to clear
them out internally after I fixed the issue that I believe was
responsible for the extraneous deletes.

--
Brad


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 02:53 AM
Paul Robert Marino
 
Default Strange Disk IO issue

By the way despite the recomendation against reducing the timer on cleaning out the tombstones* I would do it any way.

My experience with many other database platforms is that if the cleanups of stale entries is taking too long it means you aren't doing them often enough for your environment.

If you do a lot of deletes more often than most other people you need to do cleanups more often than other people no matter what the experts say.

Bulk cleanups are ment to make the process more optimal and allow for role back of changes but they aren't always practical for every implementation.



On May 16, 2012 10:07 PM, "Rich Megginson" <rmeggins@redhat.com> wrote:
On 05/16/2012 07:48 PM, Brad Schuetz wrote:


On 05/16/2012 06:24 PM, Rich Megginson wrote:


On 05/16/2012 06:48 PM, Brad Schuetz wrote:


Is there any way that I can remove the nsTombstone entries from the

master server so I can get this under control? *I think I found out why

I have so many nsTombstone entries in the first place so I'd like to get

the current ones deleted and see how much delete activity I'll have

moving forward.



I saw this bug report,

<https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to

indicate that I should be able to ldapdelete the nsTombstone entries

using their full dn, however it always says object not found.


Can you provide

1) the exact ldapdelete command line

2) the excerpts from the access and errors log from around the time of

the deletion


Here's the command with only the username part of the DN redacted and

the access log for the command, I confirmed that after the delete failed

that the entry was in fact still there by running the ldap search again

for nstombstone entries to be sure that it wasn't just a timing issue.

Also attempted the same command with quotes around the DN to delete.

The error log produced no entries around the time of the command being run.



ldapdelete -x -D 'cn=Directory Manager' -W

nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,d c=omnis,dc=com



This was the output from the above command:



ldap_delete: No such object (32)

* * * * matched DN:

uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com


Looks like 617862 is back . . .




access log:

17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from

::1 to ::1

[17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES

[17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory

Manager" method=128 version=3

[17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97

nentries=0 etime=0 dn="cn=directory manager"

[17/May/2012:01:41:35 +0000] conn=947 op=1 DEL

dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"

[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107

nentries=0 etime=0

[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107

nentries=0 etime=0

[17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND

[17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1



However I can report that the internal process seems to have finally

cleaned up the bulk of the entries, I had over 20k entries on the master

and it's finally down to less than 100. *I temporarily changed the

values for when it should reap the entries to try and get it to clear

them out internally after I fixed the issue that I believe was

responsible for the extraneous deletes.



--

Brad




--

389 users mailing list

389-users@lists.fedoraproject.org

https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-17-2012, 03:30 PM
Rich Megginson
 
Default Strange Disk IO issue

On 05/16/2012 07:48 PM, Brad Schuetz wrote:

On 05/16/2012 06:24 PM, Rich Megginson wrote:

On 05/16/2012 06:48 PM, Brad Schuetz wrote:

Is there any way that I can remove the nsTombstone entries from the
master server so I can get this under control? I think I found out why
I have so many nsTombstone entries in the first place so I'd like to get
the current ones deleted and see how much delete activity I'll have
moving forward.

I saw this bug report,
<https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
indicate that I should be able to ldapdelete the nsTombstone entries
using their full dn, however it always says object not found.

Can you provide
1) the exact ldapdelete command line
2) the excerpts from the access and errors log from around the time of
the deletion

Here's the command with only the username part of the DN redacted and
the access log for the command, I confirmed that after the delete failed
that the entry was in fact still there by running the ldap search again
for nstombstone entries to be sure that it wasn't just a timing issue.
Also attempted the same command with quotes around the DN to delete.
The error log produced no entries around the time of the command being run.

ldapdelete -x -D 'cn=Directory Manager' -W
nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,d c=omnis,dc=com

This was the output from the above command:

ldap_delete: No such object (32)
matched DN:
uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com

access log:
17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
::1 to ::1
[17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
[17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
Manager" method=128 version=3
[17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
nentries=0 etime=0 dn="cn=directory manager"
[17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
[17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1

However I can report that the internal process seems to have finally
cleaned up the bulk of the entries, I had over 20k entries on the master
and it's finally down to less than 100. I temporarily changed the
values for when it should reap the entries to try and get it to clear
them out internally after I fixed the issue that I believe was
responsible for the extraneous deletes.

Good. I filed a ticket about the tombstone deletion issue:
https://fedorahosted.org/389/ticket/376

The fact that there are two result lines for the operation is a good
clue that something is going wrong.




--
Brad


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 
Old 05-18-2012, 09:10 PM
Brad Schuetz
 
Default Strange Disk IO issue

On 05/17/2012 08:30 AM, Rich Megginson wrote:
> On 05/16/2012 07:48 PM, Brad Schuetz wrote:
>> On 05/16/2012 06:24 PM, Rich Megginson wrote:
>>> On 05/16/2012 06:48 PM, Brad Schuetz wrote:
>>>> Is there any way that I can remove the nsTombstone entries from the
>>>> master server so I can get this under control? I think I found out
>>>> why
>>>> I have so many nsTombstone entries in the first place so I'd like
>>>> to get
>>>> the current ones deleted and see how much delete activity I'll have
>>>> moving forward.
>>>>
>>>> I saw this bug report,
>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=617862>, that seems to
>>>> indicate that I should be able to ldapdelete the nsTombstone entries
>>>> using their full dn, however it always says object not found.
>>> Can you provide
>>> 1) the exact ldapdelete command line
>>> 2) the excerpts from the access and errors log from around the time of
>>> the deletion
>> Here's the command with only the username part of the DN redacted and
>> the access log for the command, I confirmed that after the delete failed
>> that the entry was in fact still there by running the ldap search again
>> for nstombstone entries to be sure that it wasn't just a timing issue.
>> Also attempted the same command with quotes around the DN to delete.
>> The error log produced no entries around the time of the command
>> being run.
>>
>> ldapdelete -x -D 'cn=Directory Manager' -W
>> nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,d c=omnis,dc=com
>>
>>
>> This was the output from the above command:
>>
>> ldap_delete: No such object (32)
>> matched DN:
>> uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com
>>
>> access log:
>> 17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
>> ::1 to ::1
>> [17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
>> [17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
>> Manager" method=128 version=3
>> [17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
>> nentries=0 etime=0 dn="cn=directory manager"
>> [17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
>> dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
>>
>> [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>> nentries=0 etime=0
>> [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>> nentries=0 etime=0
>> [17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
>> [17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1
>>
>> However I can report that the internal process seems to have finally
>> cleaned up the bulk of the entries, I had over 20k entries on the master
>> and it's finally down to less than 100. I temporarily changed the
>> values for when it should reap the entries to try and get it to clear
>> them out internally after I fixed the issue that I believe was
>> responsible for the extraneous deletes.
> Good. I filed a ticket about the tombstone deletion issue:
> https://fedorahosted.org/389/ticket/376
>
> The fact that there are two result lines for the operation is a good
> clue that something is going wrong.
>

Looks like my IO issue is resolved now that the tombstone entries aren't
going crazy.

Thanks for the help everyone.

--
Brad
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users
 

Thread Tools




All times are GMT. The time now is 11:07 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org