On thursday I was about to upgrade apache-2.2.16 to -2.2.17.
It compiled flawlessly as always. However after I restarted the daemon the
ssl connections timed out. I tried to revert the installation to the
previous version, but the symptoms remained.

I had to restore apache from my backup.

Now I'm stuck.

The linking seems to be the same:
correct module
ldd /usr/lib/apache2/modules/mod_ssl.so
linux-gate.so.1 => (0x4f33b000)
libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x4f287000)
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x4f0fb000)
libgmp.so.3 => /usr/lib/libgmp.so.3 (0x4f0aa000)
libdl.so.2 => /lib/libdl.so.2 (0x4f0a6000)
libz.so.1 => /lib/libz.so.1 (0x4f08f000)
libpthread.so.0 => /lib/libpthread.so.0 (0x4f075000)
libc.so.6 => /lib/libc.so.6 (0x4ef16000)
/lib/ld-linux.so.2 (0x4f33c000)
incorrect module
ldd /usr/lib/apache2/modules/mod_ssl.so
linux-gate.so.1 => (0x4c38c000)
libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x4c2d7000)
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x4c14b000)
libgmp.so.3 => /usr/lib/libgmp.so.3 (0x4c0fa000)
libdl.so.2 => /lib/libdl.so.2 (0x4c0f6000)
libz.so.1 => /lib/libz.so.1 (0x4c0df000)
libpthread.so.0 => /lib/libpthread.so.0 (0x4c0c5000)
libc.so.6 => /lib/libc.so.6 (0x4bf66000)
/lib/ld-linux.so.2 (0x4c38d000)

Here is an exmaple of two a PAX terminations:
Apr 17 01:47:51 atoth kernel: PAX: From 66.249.71.137: execution attempt
in: (null), 00000000-00000000 00000000
Apr 17 01:47:51 atoth kernel: PAX: terminating task:
/usr/sbin/apache2(apache2):3531, uid/euid: 81/81, PC: 00000058, SP:
484c1a7c
Apr 17 01:47:51 atoth kernel: PAX: bytes at PC: ?? ?? ?? ?? ?? ?? ?? ?? ??
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
Apr 17 01:47:51 atoth kernel: PAX: bytes at SP-4: 484c1b18 4e5c60f4
158393c4 484c1af8 484c1af4 00000000 4e83b317 4e5de8c8 4e83c7b9 4e5d52a2
155058f0 484c1b08 00000dcb 07fc8be9 00000001 4e50c07f 484c1ae8 4e525980
00000001 484c1af8 484c1af4
Apr 17 01:47:51 atoth kernel: PAX: From 66.249.71.137: execution attempt
in: (null), 00000000-00000000 00000000
Apr 17 01:47:51 atoth kernel: PAX: terminating task:
/usr/sbin/apache2(apache2):3554, uid/euid: 81/81, PC: 00000058, SP:
484c1d2c
Apr 17 01:47:51 atoth kernel: PAX: bytes at PC: ?? ?? ?? ?? ?? ?? ?? ?? ??
?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
Apr 17 01:47:51 atoth kernel: PAX: bytes at SP-4: 484c1dc8 4e5c60f4
158393c4 484c1da8 484c1da4 00000000 0000000b 00000000 484c1da8 4e3e314b
00004458 4e57a7d9 0000029c 0000000b 0000000a 0000000c 4e57a7d9 0000029a
0000000b 484c1da8 484c1da4

The linking consistency is OK. Revdep-ebuild and lafilefixer --justfixit
finds no packages to recompile.
But my current toolchain still produces unusable apache packages.
Reverting to the old binary makes the problem go away.

Portage 2.1.9.42
hardened/linux/x86
gcc-4.5.2
glibc-2.13-r2
2.6.38-hardened
gentoo-1.12.14
apache-2.2.16
openssl-1.0.0d
openssh-5.8_p1-r1

I couldn't find any other useful messages in the log.
How I should continue tracking down the problem?

Please help me:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

Hi!

On Sun, Apr 17, 2011 at 02:17:21AM +0200, "Tóth Attila" wrote:
> Reverting to the old binary makes the problem go away.

Any chance it's as trivial as somehow modified old binary - like with paxctl?

Also, you can try to use non-hardened gcc to build apache, just in case.

--
WBR, Alex.

2011.Ã�prilis 17.(V) 03:49 idÅ‘pontban Alex Efros ezt Ã*rta:
> Hi!
>
> On Sun, Apr 17, 2011 at 02:17:21AM +0200, "Tóth Attila" wrote:
>> Reverting to the old binary makes the problem go away.
>
> Any chance it's as trivial as somehow modified old binary - like with
> paxctl?

paxctl -m haven't solved the problem.

>
> Also, you can try to use non-hardened gcc to build apache, just in case.

I would rather not use a non-hardened apache on the server. But I can give
a try to compile it using a vanilla gcc profile.
Any of you successfully recompiled apache with a recent toolchain and see
the ssl connections are working correctly?

Thx:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

>
> --
> WBR, Alex.
>
>

On 17 Apr 2011 at 12:27, "Tóth Attila" wrote:

> 2011.Április 17.(V) 03:49 id"opontban Alex Efros ezt írta:
> > Hi!
> >
> > On Sun, Apr 17, 2011 at 02:17:21AM +0200, "Tóth Attila" wrote:
> >> Reverting to the old binary makes the problem go away.
> >
> > Any chance it's as trivial as somehow modified old binary - like with
> > paxctl?
>
> paxctl -m haven't solved the problem.

did you try to debug it live or look at the coredump? knowning the stack
backtrace would be useful to know who ended up calling a null funtion ptr...

söndag 17 april 2011 12.27.19 skrev Tóth Attila:
> 2011.Ã�prilis 17.(V) 03:49 idÅ‘pontban Alex Efros ezt Ã*rta:
> > Hi!
> >
> > On Sun, Apr 17, 2011 at 02:17:21AM +0200, "Tóth Attila" wrote:
> >> Reverting to the old binary makes the problem go away.
> >
> > Any chance it's as trivial as somehow modified old binary - like with
> > paxctl?
>
> paxctl -m haven't solved the problem.
>
> > Also, you can try to use non-hardened gcc to build apache, just in case.
>
> I would rather not use a non-hardened apache on the server. But I can give
> a try to compile it using a vanilla gcc profile.
> Any of you successfully recompiled apache with a recent toolchain and see
> the ssl connections are working correctly?
>
> Thx:
> Dw.
>
> > --
> >
> > WBR, Alex.
Look at bug http://bugs.gentoo.org/show_bug.cgi?id=363443
/Magnus

2011.Ã�prilis 17.(V) 13:20 idÅ‘pontban Magnus Granberg ezt Ã*rta:
> söndag 17 april 2011 12.27.19 skrev Tóth Attila:
>> 2011.Ã�prilis 17.(V) 03:49 idÅ‘pontban Alex Efros ezt Ã*rta:
>> > Hi!
>> >
>> > On Sun, Apr 17, 2011 at 02:17:21AM +0200, "Tóth Attila" wrote:
>> >> Reverting to the old binary makes the problem go away.
>> >
>> > Any chance it's as trivial as somehow modified old binary - like with
>> > paxctl?
>>
>> paxctl -m haven't solved the problem.
>>
>> > Also, you can try to use non-hardened gcc to build apache, just in
>> case.
>>
>> I would rather not use a non-hardened apache on the server. But I can
>> give
>> a try to compile it using a vanilla gcc profile.
>> Any of you successfully recompiled apache with a recent toolchain and
>> see
>> the ssl connections are working correctly?
>>
>> Thx:
>> Dw.
>>
>> > --
>> >
>> > WBR, Alex.
> Look at bug http://bugs.gentoo.org/show_bug.cgi?id=363443
> /Magnus

Compiling using gcc-4.5.2 with -O1 or switching to gcc-4.4.5 solves the
issue. Obviously it's not a solution.
I can provide binaries, but gcc cannot compile using -g ggdb in my case.

Thx for the tip. I add my comment to this bug.

Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057