Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 2010-11-23 at 11:23 +0100, Michal Schmidt wrote:
> On Tue, 23 Nov 2010 11:11:13 +0100 Patrick MONNERAT wrote:
> > While applying today's updates on a machine running a slapd server,
> > the following error occurred: [...]
>
> Have you reported it as a bug against the openldap package?
Not yet: I'm really busy trying to restore the production environment...
> Have you tried yum downgrade to the previous version?
I've just done it: seems successful at first glance.
>
> Michal


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On 23/11/10 10:11, Patrick MONNERAT wrote:
> While applying today's updates on a machine running a slapd server, the
> following error occurred:
>
> Stopping slapd: [ OK ]
> Checking configuration files for slapd: [FAILED]
> bdb(dc=linuxdev,dc=datasphere,dc=ch): Build signature doesn't match
> environment
> bdb_db_open: database "dc=linuxdev,dc=datasphere,dc=ch" cannot be
> opened, err -30971. Restore from backup!
> backend_startup_one (type=bdb,
> suffix="dc=linuxdev,dc=datasphere,dc=ch"): bi_db_open failed! (-30971)
> slap_startup failed (test would succeed using the -u switch)
> stale lock files may be present in /var/lib/ldap[WARNING]
> /var/lib/ldap /
> /
>
> as a result, the ldap server is not running anymore, I cannot start it
> manually and I have no recent backup.
>
> I cannot even use slapcat (after update) on the current data.
>
> This is quite urgent since ldap data are heavily used by our
> applications.
> Please help !

Just had the same thing happen to me.

Worked around it by doing:

# yum downgrade openldap openldap-servers openldap-clients
# slapcat > my.ldif
# yum update openldap openldap-servers openldap-clients

Remove contents of /var/lib/ldap except DB_CONFIG

# slapadd < my.ldif
# chown ldap:ldap /var/lib/ldap/*
# restorecon -rvF /var/lib/ldap
# service slapd start

It came back up OK.

Looks like the new openldap is built against a different BerkeleyDB than
the old one.

Paul.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 23 Nov 2010, Paul Howarth wrote:

> On 23/11/10 10:11, Patrick MONNERAT wrote:
>> While applying today's updates on a machine running a slapd server, the
>> following error occurred:
>>
>> Stopping slapd: [ OK ]
>> Checking configuration files for slapd: [FAILED]
>> bdb(dc=linuxdev,dc=datasphere,dc=ch): Build signature doesn't match
>> environment
>> bdb_db_open: database "dc=linuxdev,dc=datasphere,dc=ch" cannot be
>> opened, err -30971. Restore from backup!
>> backend_startup_one (type=bdb,
>> suffix="dc=linuxdev,dc=datasphere,dc=ch"): bi_db_open failed! (-30971)
>> slap_startup failed (test would succeed using the -u switch)
>> stale lock files may be present in /var/lib/ldap[WARNING]
>> /var/lib/ldap /
>> /
>>
>> as a result, the ldap server is not running anymore, I cannot start it
>> manually and I have no recent backup.
>>
>> I cannot even use slapcat (after update) on the current data.
>>
>> This is quite urgent since ldap data are heavily used by our
>> applications.
>> Please help !
>
> Just had the same thing happen to me.
>
> Worked around it by doing:
>
> # yum downgrade openldap openldap-servers openldap-clients
> # slapcat > my.ldif
> # yum update openldap openldap-servers openldap-clients
>
> Remove contents of /var/lib/ldap except DB_CONFIG
>
> # slapadd < my.ldif
> # chown ldap:ldap /var/lib/ldap/*
> # restorecon -rvF /var/lib/ldap
> # service slapd start
>
> It came back up OK.
>
> Looks like the new openldap is built against a different BerkeleyDB than
> the old one.

Yup, Berkeley DB is picky about its environment. It should be sufficient
to to do 'rm -f /var/lib/ldap/__db.*; service slapd start' to recover
from the upgrade.

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 23 Nov 2010, Panu Matilainen wrote:

> On Tue, 23 Nov 2010, Paul Howarth wrote:
>
>> On 23/11/10 10:11, Patrick MONNERAT wrote:
>>> While applying today's updates on a machine running a slapd server, the
>>> following error occurred:
>>>
>>> Stopping slapd: [ OK ]
>>> Checking configuration files for slapd: [FAILED]
>>> bdb(dc=linuxdev,dc=datasphere,dc=ch): Build signature doesn't match
>>> environment
>>> bdb_db_open: database "dc=linuxdev,dc=datasphere,dc=ch" cannot be
>>> opened, err -30971. Restore from backup!
>>> backend_startup_one (type=bdb,
>>> suffix="dc=linuxdev,dc=datasphere,dc=ch"): bi_db_open failed! (-30971)
>>> slap_startup failed (test would succeed using the -u switch)
>>> stale lock files may be present in /var/lib/ldap[WARNING]
>>> /var/lib/ldap /
>>> /
>>>
>>> as a result, the ldap server is not running anymore, I cannot start it
>>> manually and I have no recent backup.
>>>
>>> I cannot even use slapcat (after update) on the current data.
>>>
>>> This is quite urgent since ldap data are heavily used by our
>>> applications.
>>> Please help !
>>
>> Just had the same thing happen to me.
>>
>> Worked around it by doing:
>>
>> # yum downgrade openldap openldap-servers openldap-clients
>> # slapcat > my.ldif
>> # yum update openldap openldap-servers openldap-clients
>>
>> Remove contents of /var/lib/ldap except DB_CONFIG
>>
>> # slapadd < my.ldif
>> # chown ldap:ldap /var/lib/ldap/*
>> # restorecon -rvF /var/lib/ldap
>> # service slapd start
>>
>> It came back up OK.
>>
>> Looks like the new openldap is built against a different BerkeleyDB than
>> the old one.
>
> Yup, Berkeley DB is picky about its environment. It should be sufficient
> to to do 'rm -f /var/lib/ldap/__db.*; service slapd start' to recover
> from the upgrade.

...but just in case: I'm not at all familiar with openldap specifics.
The openldap maintenance guide has the correct procedure for upgrades:
http://www.openldap.org/doc/admin24/maintenance.html (which is basically
the steps explained by Paul above)

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

Hi!

Currently, the upgrade process in openldap looks like this:
* during db4 package upgrade run db_upgrade (%triggerin and %triggerun)
* if minor version of openldap changes (e.g. 2.3 -> 2.4), export the database,
delete it and import it back (which is recommended by maintanence guide, as
Panu mentioned)

We didn't wanted to do the export+import during each upgrade, as it takes
quite a long time if you have large database. But it seems that current
process doesn't work and that doing it every time will be the safest way.
(Maybe we can ignore epoch changes.)

Thanks for suggestions. I will fix it today and push an update.

Patric, thank you for reporting this. And sorry for the difficulties.

Jan

--
Jan V�elák
Base Operating Systems Brno
Red Hat Inc.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 2010-11-23 at 14:55 +0100, Jan Vcelak wrote:

> Patric, thank you for reporting this. And sorry for the difficulties.

You're welcome. And never mind for the difficulties: I understand your
trouble and I wouldn't be in such a sh... myself !

I succeeded in restoring the LDAP usability by downgrading to 2.4.22-7,
but I have to wait until after 16:00 UTC for the DB migration (after
production use stops).

Many thanks to all who helped.

Cheers,
Patrick

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

Related question to other developers:

Am I allowed to write some additional output in %pre/%post scriptlets? And
shall I use stdout or stderr? I haven't found it in guidelines.

This is the problem: The database migration could take a really long time. I
have testing data with 56 entries (nodes) - exporting (slapcat) is quite fast,
but importing (slapadd) takes around 10 seconds.

Imagine you have a large database. I would like to inform the user, that the
database migration is in progress. Do you have some ideas?

Jan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 2010-11-23 at 17:09 +0100, Jan Vcelak wrote:
> This is the problem: The database migration could take a really long time. I
> have testing data with 56 entries (nodes) - exporting (slapcat) is quite fast,
> but importing (slapadd) takes around 10 seconds.
>
> Imagine you have a large database. I would like to inform the user, that the
> database migration is in progress. Do you have some ideas?

I would have thought this was exactly the kind of task that shouldn't
happen during a stable release?

Cheers

Alex.


--
This message was scanned by Better Hosted and is believed to be clean.
http://www.betterhosted.com

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On Tue, 2010-11-23 at 17:09 +0100, Jan Vcelak wrote:

> Am I allowed to write some additional output in %pre/%post scriptlets? And
> shall I use stdout or stderr? I haven't found it in guidelines.

I was always told "no output allowed", which is sad. Sure, it might not
be seen in all cases. What would be cool is if there were some way to
indicate to RPM that progress was being made (some kind of incrementing
counter, or whatever) that could be pushed up to higher level stuff,
like PackageKit. I don't think there's anything today.

Jon.


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Urgent: today's F14 catastrophe with openldap-servers update
 

On 11/23/10 5:55 AM, Jan Vcelak wrote:
> Hi!
>
> Currently, the upgrade process in openldap looks like this:
> * during db4 package upgrade run db_upgrade (%triggerin and %triggerun)
> * if minor version of openldap changes (e.g. 2.3 -> 2.4), export the database,
> delete it and import it back (which is recommended by maintanence guide, as
> Panu mentioned)
>
> We didn't wanted to do the export+import during each upgrade, as it takes
> quite a long time if you have large database. But it seems that current
> process doesn't work and that doing it every time will be the safest way.
> (Maybe we can ignore epoch changes.)
>
> Thanks for suggestions. I will fix it today and push an update.
>
> Patric, thank you for reporting this. And sorry for the difficulties.

Why was this update made on F14 in the first place?

--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel