On 7 Mar 2009 at 18:39, basile wrote:

> Hi guys,
>
> I'm encountering a reproduceable problem with hardened 2.6.26-r9 and
> 2.6.27-r8 that wasn't there with 2.6.25-r13 on i686, and isn't there
> with amd64 using approximately the same kernel configuration in every
> case. I've been able to reproduce it in vmware, qemu and on physical
> boxes, one with a Intel(R) Core(TM)2 Quad CPU Q6700, the other AMD
> Athlon(tm) 64 FX-62 Dual Core. It a stack fault in kernel mode, but I
> can't pin it down further. It happens almost immediately after the
> bootloader passes control to the kernel. The best error message comes
> from qemu which gives the states of the registers. Here's the error
> message from a bootable ISO I made suing 2.6.26-r9. Any idea where I
> can start tackling this one?

you'll have to check what code was executed just before the triple fault,
start at around EIP. also passing -d in_asm,int,exec,cpu,pcall will produce
a nice log file that will make it even easier.

>
> # qemu -cdrom th-i686-20090307-RC3.iso
> qemu: fatal: triple fault
> EAX=000000ff EBX=0153cac0 ECX=0013a2d1 EDX=0013a2d1
> ESI=0024c000 EDI=01400000 EBP=01541a20 ESP=01541a10
> EIP=0153a2d0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
^^^^^^^^^^^^

Hi guys,

I'm encountering a reproduceable problem with hardened 2.6.26-r9 and
2.6.27-r8 that wasn't there with 2.6.25-r13 on i686, and isn't there
with amd64 using approximately the same kernel configuration in every
case. I've been able to reproduce it in vmware, qemu and on physical
boxes, one with a Intel(R) Core(TM)2 Quad CPU Q6700, the other AMD
Athlon(tm) 64 FX-62 Dual Core. It a stack fault in kernel mode, but I
can't pin it down further. It happens almost immediately after the
bootloader passes control to the kernel. The best error message comes
from qemu which gives the states of the registers. Here's the error
message from a bootable ISO I made suing 2.6.26-r9. Any idea where I
can start tackling this one?

# qemu -cdrom th-i686-20090307-RC3.iso
qemu: fatal: triple fault
EAX=000000ff EBX=0153cac0 ECX=0013a2d1 EDX=0013a2d1
ESI=0024c000 EDI=01400000 EBP=01541a20 ESP=01541a10
EIP=0153a2d0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 00000000 ffffffff 00cf9300
CS =0010 00000000 ffffffff 00cf9b00
SS =0018 00000000 ffffffff 00cf9300
DS =0018 00000000 ffffffff 00cf9300
FS =0018 00000000 ffffffff 00cf9300
GS =0018 00000000 ffffffff 00cf9300
LDT=0000 00000000 00000000 00008000
TR =0020 00001000 00000067 00008900
GDT= 000928f0 00000027
IDT= 00000000 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
CCS=0024c000 CCD=ffeee2d1 CCO=SUBL
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
Aborted

--

Anthony G. Basile, Ph.D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
USA

(716) 829-8197

On 8 Mar 2009 at 18:51, basile wrote:

> >> # qemu -cdrom th-i686-20090307-RC3.iso
> >> qemu: fatal: triple fault
> >> EAX=000000ff EBX=0153cac0 ECX=0013a2d1 EDX=0013a2d1
> >> ESI=0024c000 EDI=01400000 EBP=01541a20 ESP=01541a10
> >> EIP=0153a2d0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> >>
> > ^^^^^^^^^^^^
> >
> >
> 1) Thanks for the suggestion with -d in_asm,int,exec,cpu,pcall Very
> useful.

if you upload that log and the corresponding vmlinux or just the iso itself
somewhere, i can take a look, otherwise it's pretty much impossible to tell
what caused the triple fault. would also be nice if you tried .28 before
anything else, just to be sure it's not something i'd already fixed.

pageexec@freemail.hu wrote:
> On 7 Mar 2009 at 18:39, basile wrote:
>
>
>> Hi guys,
>>
>> I'm encountering a reproduceable problem with hardened 2.6.26-r9 and
>> 2.6.27-r8 that wasn't there with 2.6.25-r13 on i686, and isn't there
>> with amd64 using approximately the same kernel configuration in every
>> case. I've been able to reproduce it in vmware, qemu and on physical
>> boxes, one with a Intel(R) Core(TM)2 Quad CPU Q6700, the other AMD
>> Athlon(tm) 64 FX-62 Dual Core. It a stack fault in kernel mode, but I
>> can't pin it down further. It happens almost immediately after the
>> bootloader passes control to the kernel. The best error message comes
>> from qemu which gives the states of the registers. Here's the error
>> message from a bootable ISO I made suing 2.6.26-r9. Any idea where I
>> can start tackling this one?
>>
>
> you'll have to check what code was executed just before the triple fault,
> start at around EIP. also passing -d in_asm,int,exec,cpu,pcall will pro
> a nice log file that will make it even easier.
>
>
>> # qemu -cdrom th-i686-20090307-RC3.iso
>> qemu: fatal: triple fault
>> EAX=000000ff EBX=0153cac0 ECX=0013a2d1 EDX=0013a2d1
>> ESI=0024c000 EDI=01400000 EBP=01541a20 ESP=01541a10
>> EIP=0153a2d0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>
> ^^^^^^^^^^^^
>
>
1) Thanks for the suggestion with -d in_asm,int,exec,cpu,pcall Very
useful.

2) I've localized the problem, but I can't figure out what's changed
from 2.6.25 to 2.6.26/2.6.27. The triple fault occurs when activating
CONFIG_PAX_KERNEXEC in the later two, but poses no problem in 2.6.25.
Comparing the source trees for 2.6.25-hardened-r13 and
2.6.26-hardened-r9, I found no clues tracing back from places where
#ifdef CONFIG_PAX_KERNEXEC includes code. I compared the patches
(4420_grsec) and the only "suspicious" changes I saw were 1) in
arch/x86/kernel/module_32.c where invoking vmalloc changed and 2) in
arch/x86/kernel/head64.c (2.6.25) -> arch/x86/kernel/head_32.S (2.6.26)
there is some inline assembly which changed. I don't really understand
these two changes yet.

Any more suggestions? I can always disable the option, but I like the
feature.

--

Anthony G. Basile, Ph.D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
USA

(716) 829-8197

On 8 Mar 2009 at 20:10, basile wrote:

> I built a grub ISO with two kernels, identical in every respect except
> CONFIG_PAX_KERNEXEC is set for the first kernel which gives the triple
> fault, and not set for the second which boots fine. Here's the ISO and
> qemu.log:
>
> http://opensource.dyc.edu/pub/misc/ramdisk.iso
> http://opensource.dyc.edu/pub/misc/qemu.log

hmm, it looks like an early decompression problem, something that a bad ld
would produce. what's your binutils version? pax needs 2.18+ these days.

On 8 Mar 2009 at 20:47, basile wrote:

> pageexec@freemail.hu wrote:
> > On 8 Mar 2009 at 20:10, basile wrote:
> >
> >
> >> I built a grub ISO with two kernels, identical in every respect except
> >> CONFIG_PAX_KERNEXEC is set for the first kernel which gives the triple
> >> fault, and not set for the second which boots fine. Here's the ISO and
> >> qemu.log:
> >>
> >> http://opensource.dyc.edu/pub/misc/ramdisk.iso
> >> http://opensource.dyc.edu/pub/misc/qemu.log
> >>
> >
> > hmm, it looks like an early decompression problem, something that a bad ld
> > would produce. what's your binutils version? pax needs 2.18+ these days.
> >
> >
>
> binutils-2.18-r3

hmm, that used to be ok, can you try 2.19.x just in case?

pageexec@freemail.hu wrote:
> On 8 Mar 2009 at 18:51, basile wrote:
>
>
>>>> # qemu -cdrom th-i686-20090307-RC3.iso
>>>> qemu: fatal: triple fault
>>>> EAX=000000ff EBX=0153cac0 ECX=0013a2d1 EDX=0013a2d1
>>>> ESI=0024c000 EDI=01400000 EBP=01541a20 ESP=01541a10
>>>> EIP=0153a2d0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>
>>>>
>>> ^^^^^^^^^^^^
>>>
>>>
>>>
>> 1) Thanks for the suggestion with -d in_asm,int,exec,cpu,pcall Very
>> useful.
>>
>
> if you upload that log and the corresponding vmlinux or just the iso itself
> somewhere, i can take a look, otherwise it's pretty much impossible to tell
> what caused the triple fault. would also be nice if you tried .28 before
> anything else, just to be sure it's not something i'd already fixed.
>
>

I built a grub ISO with two kernels, identical in every respect except
CONFIG_PAX_KERNEXEC is set for the first kernel which gives the triple
fault, and not set for the second which boots fine. Here's the ISO and
qemu.log:

http://opensource.dyc.edu/pub/misc/ramdisk.iso
http://opensource.dyc.edu/pub/misc/qemu.log

The second kernel boots into a busybox environment if you want to look
at /proc/config.gz

I'll try .28

--

Anthony G. Basile, Ph.D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
USA

(716) 829-8197

pageexec@freemail.hu wrote:
> On 8 Mar 2009 at 20:10, basile wrote:
>
>
>> I built a grub ISO with two kernels, identical in every respect except
>> CONFIG_PAX_KERNEXEC is set for the first kernel which gives the triple
>> fault, and not set for the second which boots fine. Here's the ISO and
>> qemu.log:
>>
>> http://opensource.dyc.edu/pub/misc/ramdisk.iso
>> http://opensource.dyc.edu/pub/misc/qemu.log
>>
>
> hmm, it looks like an early decompression problem, something that a bad ld
> would produce. what's your binutils version? pax needs 2.18+ these days.
>
>

binutils-2.18-r3

Gentoo hardened toolchain.

--

Anthony G. Basile, Ph.D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
USA

(716) 829-8197

pageexec@freemail.hu wrote:
> On 8 Mar 2009 at 20:47, basile wrote:
>
>
>> pageexec@freemail.hu wrote:
>>
>>> On 8 Mar 2009 at 20:10, basile wrote:
>>>
>>>
>>>
>>>> I built a grub ISO with two kernels, identical in every respect except
>>>> CONFIG_PAX_KERNEXEC is set for the first kernel which gives the triple
>>>> fault, and not set for the second which boots fine. Here's the ISO and
>>>> qemu.log:
>>>>
>>>> http://opensource.dyc.edu/pub/misc/ramdisk.iso
>>>> http://opensource.dyc.edu/pub/misc/qemu.log
>>>>
>>>>
>>> hmm, it looks like an early decompression problem, something that a bad ld
>>> would produce. what's your binutils version? pax needs 2.18+ these days.
>>>
>>>
>>>
>> binutils-2.18-r3
>>
>
> hmm, that used to be ok, can you try 2.19.x just in case?
>
>
Okay tested vanilla binutils-2.19.1 Same behaviour.

I'll test hardened 2.6.28 tomorrow. Too tired tonight.

--

Anthony G. Basile, Ph.D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
USA

(716) 829-8197

On 8 Mar 2009 at 22:42, basile wrote:

> > hmm, that used to be ok, can you try 2.19.x just in case?
> >
> >
> Okay tested vanilla binutils-2.19.1 Same behaviour.
>
> I'll test hardened 2.6.28 tomorrow. Too tired tonight.

no worries, i already reproduced it with latest PaX and everything and debugged it
a bit. it's the early decompressor phase that somehow manages to overwrite itself as
well and that causes an invalid opcode and further exceptions when the decompressor
tries to print out a message (the putstr() code happens to be destroyed by then).
this is likely due to some code underestimating the needed size of the decompressed
stream, but i don't yet know why it's config dependent and whether it's some linker
issue or a bug in PaX.