FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Hardened

 
 
LinkBack Thread Tools
 
Old 03-06-2009, 02:57 AM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

It's ~6am here and I'm too tired to file new bugs, I wanna sleep a little first.

While in general Gentoo "stable" branch is very stable, shit always happens.
This time is was upgrade from 2.6.26-hardened-r9 to 2.6.27-hardened-r8.

First issue: many perl scripts (including FastCGI servers) failed to start
with segmentation fault. See http://bugs.gentoo.org/show_bug.cgi?id=261357
for details and ugly workarounds.

Second issue: apache failed to start with segmentation fault. As I said,
I'll file bugs later, but in short it's trouble with Ioncube and
ZendOptimizer. I had to switch off both to run apache. I've no idea how it
may be possible to workaround this without doing paxctl -m on apache.


For now I just rollback to previous kernel (I think that will be more
secure than paxctl -m for apache, plus I afraid new kernel may do other
nasty things too).


Resume: this upgrade kill both perl and apache.

--
WBR, Alex.
 
Old 03-06-2009, 06:11 AM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

On Fri, Mar 06, 2009 at 05:57:18AM +0200, Alex Efros wrote:
> For now I just rollback to previous kernel (I think that will be more
> secure than paxctl -m for apache, plus I afraid new kernel may do other
> nasty things too).
>
>
> Resume: this upgrade kill both perl and apache.

And it just killed amarok on my workstation, forcing me to downgrade
kernel on workstation too.

--
WBR, Alex.
 
Old 03-06-2009, 06:15 AM
 
Default 2.6.27-hardened-r8: assassination

On 6 Mar 2009 at 5:57, Alex Efros wrote:

> First issue: many perl scripts (including FastCGI servers) failed to start
> with segmentation fault. See http://bugs.gentoo.org/show_bug.cgi?id=261357
> for details and ugly workarounds.
>
> Second issue: apache failed to start with segmentation fault. As I said,
> I'll file bugs later, but in short it's trouble with Ioncube and
> ZendOptimizer. I had to switch off both to run apache. I've no idea how it
> may be possible to workaround this without doing paxctl -m on apache.

two things i'd like you to try:

1. 2.6.28.7 and PaX alone
2. get coredumps and analyze them for the usual things, to see why the segfaults
occured. if that doesn't point to anything, maybe try an strace as well.
 
Old 03-06-2009, 02:13 PM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

On Fri, Mar 06, 2009 at 09:15:36AM +0200, pageexec@freemail.hu wrote:
> two things i'd like you to try:
>
> 1. 2.6.28.7 and PaX alone
> 2. get coredumps and analyze them for the usual things, to see why the segfaults
> occured. if that doesn't point to anything, maybe try an strace as well.

Two questions:
1) Is "2.6.28.7 and PaX alone" mean hardened-sources-2.6.28 with
everything except PaX switched off, or vanilla-sources-2.6.28.7 manually
patched with latests PaX?
2) I'm perl programmer, not C. So I need more detailed instructions (list
of commands to run) how to "get coredumps and analyze them for the usual
things". Probably this info already available somewhere, so url to this
doc will be enough.

As for strace - did that, it helps me detect .so libraries (Ioncube and
ZendOptimizer) because of which apache was killed.

--
WBR, Alex.
 
Old 03-06-2009, 04:28 PM
 
Default 2.6.27-hardened-r8: assassination

On 6 Mar 2009 at 17:13, Alex Efros wrote:

> Two questions:
> 1) Is "2.6.28.7 and PaX alone" mean hardened-sources-2.6.28 with
> everything except PaX switched off, or vanilla-sources-2.6.28.7 manually
> patched with latests PaX?

it's always the latter , i need to make sure it's a PaX problem.

> 2) I'm perl programmer, not C. So I need more detailed instructions (list
> of commands to run) how to "get coredumps and analyze them for the usual
> things". Probably this info already available somewhere, so url to this
> doc will be enough.

i mentioned them quite a few times on the list and bugzilla and the grsec forums,
here it is again. first, the coredump: you enable coredumps in your shell
(ulimit -c unlimited) then run your program that crashes. this will produce
a coredump file that you load into gdb and then issue the following gdb commands:

bt
x/8i $pc
x/8x $sp
info reg

> As for strace - did that, it helps me detect .so libraries (Ioncube and
> ZendOptimizer) because of which apache was killed.

on a second thought, i'd need the strace output regardless of the gdb analysis,
just to see how text relocations went as that's where the problem is probably.
 
Old 03-06-2009, 08:12 PM
 
Default 2.6.27-hardened-r8: assassination

On 6 Mar 2009 at 23:51, Alex Efros wrote:

> When I run apache for the first time after reboot - without strace/core,
> just to see is it crash - I got this in kernel log:
>
> 2009-03-06_20:48:56.60108 kern.info: apache2[4621]: segfault at
> 4d554ed0 ip 4d541399 sp 594130d0 error 7 in ld-2.6.1.so[4d53a000+1a000]

ah crap, i know what it is. it's a several years old glibc bug where someone
put a certain variable into the RELRO segment but forgot that it'll be written
to later when a library with RWE GNU_STACK is loaded. the workaround is to find
that library (just extract them from strace, probably it'll be pari's library)
and run execstack -c on it.
 
Old 03-06-2009, 08:51 PM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

On Fri, Mar 06, 2009 at 07:28:17PM +0200, pageexec@freemail.hu wrote:
> it's always the latter , i need to make sure it's a PaX problem.

Ok. With this kernel, using pax-linux-2.6.28.7-test19.patch, I was able to
reproduce issues with apache/php/{ioncube,zendoptimizer} and perl module
Math::Pari. Amarok doesn't crash.

> i mentioned them quite a few times on the list and bugzilla and the grsec forums,
> here it is again. first, the coredump: you enable coredumps in your shell

thanks for instructions, here are results:


I've tried to recompile perl, apache and php with "debug" USE-flag enabled,
but looks like ioncube&zendoptimizer don't support php built this way.
So, only perl & apache was built with "debug" flag.

When I run apache for the first time after reboot - without strace/core,
just to see is it crash - I got this in kernel log:

2009-03-06_20:48:56.60108 kern.info: apache2[4621]: segfault at
4d554ed0 ip 4d541399 sp 594130d0 error 7 in ld-2.6.1.so[4d53a000+1a000]

I must note it looks very similar to errors I got previously with this
issue - segfault always was reported like "error 7 in ld-2.6.1.so".

But all next runs (under strace and with core dumps enabled) doesn't
produce any error messages in kernel log, which is quite unusual.



# strace -f apache2 -D NO_DETACH -k start -D MANUAL -D DEFLATE -D FASTCGI -D PHP5 -D SSL &>apache2.strace
# gdb
(gdb) core /core
(no debugging symbols found)
Core was generated by `apache2 -D NO_DETACH -k start -D MANUAL -D DEFLATE -D FASTCGI -D PHP5 -D SSL'.
Program terminated with signal 11, Segmentation fault.
[New process 11835]
#0 0x4ce14399 in ?? ()
(gdb) bt
#0 0x4ce14399 in ?? ()
#1 0x4ce27000 in ?? ()
#2 0x00000ed4 in ?? ()
#3 0x00000003 in ?? ()
#4 0x00000003 in ?? ()
#5 0x00000004 in ?? ()
#6 0x00000000 in ?? ()
(gdb) x/8i $pc
0x4ce14399: Cannot access memory at address 0x4ce14399
(gdb) x/8x $sp
0x5a681770: 0x4ce27000 0x00000ed4 0x00000003 0x00000003
0x5a681780: 0x00000004 0x00000000 0x00000001 0x4cb5a170
(gdb) info reg
eax 0xffffffff -1
ecx 0x4ce27fc4 1289912260
edx 0xd 13
ebx 0x4ce27fc4 1289912260
esp 0x5a681770 0x5a681770
ebp 0x5a681890 0x5a681890
esi 0x4ce27000 1289908224
edi 0xed4 3796
eip 0x4ce14399 0x4ce14399
eflags 0x10286 [ PF SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51



# vi /etc/php/apache2-php5/php.ini ### disable ioncube
# strace -f apache2 -D NO_DETACH -k start -D MANUAL -D DEFLATE -D FASTCGI -D PHP5 -D SSL &>apache2.strace_zend
# gdb /usr/sbin/apache2 /core
This GDB was configured as "i686-pc-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
(no debugging symbols found)
Loaded symbols for /usr/sbin/apache2
...
Reading symbols from /usr/local/Zend/lib/ZendExtensionManager.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/Zend/lib/ZendExtensionManager.so

(no debugging symbols found)
Core was generated by `apache2 -D NO_DETACH -k start -D MANUAL -D DEFLATE -D FASTCGI -D PHP5 -D SSL'.
Program terminated with signal 11, Segmentation fault.
[New process 31217]
#0 0x51015399 in ?? () from /lib/ld-linux.so.2
(gdb) bt
#0 0x51015399 in ?? () from /lib/ld-linux.so.2
#1 0x51028000 in ?? ()
#2 0x00000ed4 in ?? ()
#3 0x00000003 in ?? ()
#4 0x5d5cf82c in ?? ()
#5 0x00000004 in ?? ()
#6 0x00000000 in ?? ()
(gdb) x/8i $pc
0x51015399 <free@plt+27445>: orl $0x7,-0xf4(%ebx)
0x510153a0 <free@plt+27452>: mov $0x1,%ecx
0x510153a5 <free@plt+27457>: mov %ecx,0x8(%esp)
0x510153a9 <free@plt+27461>: mov %edi,0x4(%esp)
0x510153ad <free@plt+27465>: mov %esi,(%esp)
0x510153b0 <free@plt+27468>: call 0x51022e80
0x510153b5 <free@plt+27473>: jmp 0x5101505c <free@plt+26616>
0x510153ba <free@plt+27478>: xor %ecx,%ecx
(gdb) x/8x $sp
0x5d5cf800: 0x51028000 0x00000ed4 0x00000003 0x5d5cf82c
0x5d5cf810: 0x00000004 0x00000000 0x00000001 0x50d5b170
(gdb) info reg
eax 0xffffffff -1
ecx 0x51028fc4 1359122372
edx 0xd 13
ebx 0x51028fc4 1359122372
esp 0x5d5cf800 0x5d5cf800
ebp 0x5d5cf920 0x5d5cf920
esi 0x51028000 1359118336
edi 0xed4 3796
eip 0x51015399 0x51015399 <free@plt+27445>
eflags 0x10286 [ PF SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51



# ACCEPT_KEYWORDS=~x86 emerge -a math-pari

if I run perl without strace - I got error message in kernel log:

# perl -e 'use Math::Pari;'
Segmentation fault (core dumped)

2009-03-06_21:31:02.23339 kern.info: perl[17676]: segfault at 4ebd7ed0
ip 4ebc4399 sp 58019490 error 7 in ld-2.6.1.so[4ebbd000+1a000]

if I run perl with strace - there will be no messages in kernel log

# strace -f perl -e 'use Math::Pari;' &>perl.strace
# gdb /usr/bin/perl core
This GDB was configured as "i686-pc-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
(no debugging symbols found)
Loaded symbols for /usr/bin/perl
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libcrypt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /lib/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/auto/Math/Pari/Pari.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/auto/Math/Pari/Pari.so
(no debugging symbols found)
Core was generated by `perl -e use Math::Pari;'.
Program terminated with signal 11, Segmentation fault.
[New process 30393]
#0 0x4fa55399 in ?? () from /lib/ld-linux.so.2
(gdb) bt
#0 0x4fa55399 in ?? () from /lib/ld-linux.so.2
#1 0x4fa68000 in ?? ()
#2 0x00000ed4 in ?? ()
#3 0x00000003 in ?? ()
#4 0x17364a75 in ?? () from /usr/bin/perl
#5 0x00000145 in ?? ()
#6 0x17426824 in ?? ()
#7 0x5a96a6a8 in ?? ()
#8 0x17301567 in ?? () from /usr/bin/perl
#9 0x17426824 in ?? ()
#10 0x00000050 in ?? ()
#11 0x173040d8 in Perl_av_undef () from /usr/bin/perl
#12 0x4fa55f4e in ?? () from /lib/ld-linux.so.2
#13 0x5a96a79c in ?? ()
#14 0x17443df8 in ?? ()
#15 0x00000000 in ?? ()
(gdb) x/8i $pc
0x4fa55399 <free@plt+27445>: orl $0x7,-0xf4(%ebx)
0x4fa553a0 <free@plt+27452>: mov $0x1,%ecx
0x4fa553a5 <free@plt+27457>: mov %ecx,0x8(%esp)
0x4fa553a9 <free@plt+27461>: mov %edi,0x4(%esp)
0x4fa553ad <free@plt+27465>: mov %esi,(%esp)
0x4fa553b0 <free@plt+27468>: call 0x4fa62e80
0x4fa553b5 <free@plt+27473>: jmp 0x4fa5505c <free@plt+26616>
0x4fa553ba <free@plt+27478>: xor %ecx,%ecx
(gdb) x/8x $sp
0x5a96a600: 0x4fa68000 0x00000ed4 0x00000003 0x17364a75
0x5a96a610: 0x00000145 0x17426824 0x5a96a6a8 0x17301567
(gdb) info reg
eax 0xffffffff -1
ecx 0x4fa68fc4 1336315844
edx 0xd 13
ebx 0x4fa68fc4 1336315844
esp 0x5a96a600 0x5a96a600
ebp 0x5a96a720 0x5a96a720
esi 0x4fa68000 1336311808
edi 0xed4 3796
eip 0x4fa55399 0x4fa55399 <free@plt+27445>
eflags 0x10286 [ PF SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51



> on a second thought, i'd need the strace output regardless of the gdb analysis,
> just to see how text relocations went as that's where the problem is probably.

http://powerman.name/tmp/apache2.strace
http://powerman.name/tmp/apache2.strace_zend
http://powerman.name/tmp/perl.strace

--
WBR, Alex.
 
Old 03-06-2009, 09:57 PM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

On Fri, Mar 06, 2009 at 11:12:59PM +0200, pageexec@freemail.hu wrote:
> ah crap, i know what it is. it's a several years old glibc bug where someone
> put a certain variable into the RELRO segment but forgot that it'll be written
> to later when a library with RWE GNU_STACK is loaded. the workaround is
> to find that library (just extract them from strace, probably it'll be
> pari's library) and run execstack -c on it.

I don't have execstack command. Looks like it belong to prelink package,
but http://www.gentoo.org/doc/en/prelink-howto.xml states it's
incompatible with hardened. Because of this I decide to compile it
manually, just to get execstack command:

# emerge -f prelink
# cd /usr/src
# tar xjvf /usr/portage-distfiles/prelink-20071009.tar.bz2
# cd prelink
# ./configure && make

Now I tried your workaround:

# /usr/src/prelink/src/execstack -c /usr/lib/perl5/site_perl/5.8.8/i686-linux/auto/Math/Pari/Pari.so
# /usr/src/prelink/src/execstack -c /usr/local/ioncube/ioncube_loader_lin_5.2.so
# /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/ZendExtensionManager.so
# /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/ZendExtensionManager_TS.so
# /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/Optimizer-3.3.0/php-5.2.x/ZendOptimizer.so

and it works!!

Is this issue will be fixed in next stable hardened-sources?

--
WBR, Alex.
 
Old 03-06-2009, 10:25 PM
Ned Ludd
 
Default 2.6.27-hardened-r8: assassination

On Sat, 2009-03-07 at 00:57 +0200, Alex Efros wrote:
> Hi!
>
> On Fri, Mar 06, 2009 at 11:12:59PM +0200, pageexec@freemail.hu wrote:
> > ah crap, i know what it is. it's a several years old glibc bug where someone
> > put a certain variable into the RELRO segment but forgot that it'll be written
> > to later when a library with RWE GNU_STACK is loaded. the workaround is
> > to find that library (just extract them from strace, probably it'll be
> > pari's library) and run execstack -c on it.
>
> I don't have execstack command. Looks like it belong to prelink package,
> but http://www.gentoo.org/doc/en/prelink-howto.xml states it's
> incompatible with hardened. Because of this I decide to compile it
> manually, just to get execstack command:
>
> # emerge -f prelink
> # cd /usr/src
> # tar xjvf /usr/portage-distfiles/prelink-20071009.tar.bz2
> # cd prelink
> # ./configure && make
>
> Now I tried your workaround:
>
> # /usr/src/prelink/src/execstack -c /usr/lib/perl5/site_perl/5.8.8/i686-linux/auto/Math/Pari/Pari.so
> # /usr/src/prelink/src/execstack -c /usr/local/ioncube/ioncube_loader_lin_5.2.so
> # /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/ZendExtensionManager.so
> # /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/ZendExtensionManager_TS.so
> # /usr/src/prelink/src/execstack -c /usr/local/Zend/lib/Optimizer-3.3.0/php-5.2.x/ZendOptimizer.so
>
> and it works!!
>
> Is this issue will be fixed in next stable hardened-sources?

FYI.. PaX Team maintains the PaX kernel and has little control over what
fixes go into the "next" hardened-sources. Also seems to me a little
strange that the PaX Team would have to put a work-around in the kernel
for a bug in glibc.. Seems like glibc should be fixed vs the kernel.



--
Ned Ludd <solar@gentoo.org>
Gentoo Linux
 
Old 03-06-2009, 10:46 PM
Alex Efros
 
Default 2.6.27-hardened-r8: assassination

Hi!

On Fri, Mar 06, 2009 at 03:25:16PM -0800, Ned Ludd wrote:
> FYI.. PaX Team maintains the PaX kernel and has little control over what
> fixes go into the "next" hardened-sources. Also seems to me a little
> strange that the PaX Team would have to put a work-around in the kernel
> for a bug in glibc.. Seems like glibc should be fixed vs the kernel.

Some changes in hardened-sources trigger this bug (which exists for years
and don't bother anybody) and kill my servers (apache and perl are
critical for normal server operation).

I'm neither security expert nor Gentoo developer so I don't pretend to
decide where/how this bug should be fixed and surely don't recommend to
add any work-around in the kernel (but that at a glance looks like one of
possible solutions because previous kernel don't trigger this bug).

Only thing I need to know - how long I've to live with this workaround: is
it will be fixed soon, or I have to prepare to redo these 'execstack -c'
commands in next years after each math-pari/zend/ioncube upgrade... and be
prepared to see other applications unexpectedly crash (on loading new
plugin, for example) because of this bug.

If we'll wait until "several years old glibc bug" will be fixed in glibc -
it probably will take few more years.


Anyway, thank you all for your work!
And for providing this workaround, of course - it's much better to have one.

--
WBR, Alex.
 

Thread Tools




All times are GMT. The time now is 07:11 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org