Disappearing root on 2.6.36-hardened-r6 upgrade
 

I've got (at least) two servers that lose their root partition after
this upgrade. One of them has an HP cciss SCSI RAID controller; the
other has a single IDE hard drive. Assuming the problem is something
common, I'll stick to describing the one with the array for now.

First of all, I didn't touch /etc/fstab:

/dev/cciss/c0d0p2 /boot ext3 noauto,noatime 1 2
/dev/cciss/c0d0p3 / ext4 acl,noatime 0 1
/dev/cciss/c0d0p1 none swap sw 0 0

I built the kernel after a make oldconfig, and updated grub.conf:

title Gentoo Linux 2.6.36-hardened-r6
root (hd0,1)
kernel /kernel-2.6.36-hardened-r6

It's actually there:

# /bin/ls /boot/kernel-2.6.36-hardened-r6
/boot/kernel-2.6.36-hardened-r6

But upon reboot, this happens:

http://michael.orlitzky.com/images/untouched.jpg

So, I tried it with root=/dev/cciss/c0d0p3:

http://michael.orlitzky.com/images/with_root_param.jpg

It clearly sees my partitions, since it lists them all. The root is
ext4, which is compiled into the kernel:

# grep EXT4 .config
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
# CONFIG_EXT4_FS_SECURITY is not set
# CONFIG_EXT4_DEBUG is not set

Now I'm at a loss. There must have been something else that I did during
the make oldconfig that broke it. I keep my kernel configs in git, so
here's the diff (with context stripped) from my previous kernel,
2.6.32-hardened-r22. If anyone has any ideas, I'd appreciate it:

+CONFIG_INSTRUCTION_DECODER=y
-CONFIG_GENERIC_TIME=y
+CONFIG_NEED_SG_DMA_LENGTH=y
+CONFIG_HAVE_EARLY_RES=y
+CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-ecx -fcall-saved-edx"
+CONFIG_CROSS_COMPILE=""
+CONFIG_HAVE_KERNEL_LZO=y
+CONFIG_PERF_EVENTS=y
+CONFIG_HAVE_OPTPROBES=y
+CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
+CONFIG_HAVE_HW_BREAKPOINT=y
+CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
+CONFIG_HAVE_USER_RETURN_NOTIFIER=y
+CONFIG_HAVE_PERF_EVENTS_NMI=y
+CONFIG_INLINE_SPIN_UNLOCK=y
+CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
+CONFIG_INLINE_READ_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK_IRQ=y
+CONFIG_INLINE_WRITE_UNLOCK=y
+CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
+CONFIG_MUTEX_SPIN_ON_OWNER=y
+CONFIG_NO_BOOTMEM=y
-CONFIG_X86_L1_CACHE_BYTES=64
-CONFIG_X86_INTERNODE_CACHE_BYTES=64
+CONFIG_X86_INTERNODE_CACHE_SHIFT=7
-CONFIG_HAVE_MLOCK=y
-CONFIG_HAVE_MLOCKED_PAGE_BIT=y
-CONFIG_ACPI_DOCK=y
+CONFIG_ACPI_HED=m
+CONFIG_ACPI_APEI=y
+CONFIG_ACPI_APEI_GHES=m
+CONFIG_INTEL_IDLE=y
+CONFIG_PCIEASPM=y
+CONFIG_PCI_IOAPIC=y
-CONFIG_PACKET_MMAP=y
+
+
+
+CONFIG_NETFILTER_XT_MATCH_OSF=m
-CONFIG_NETFILTER_XT_MATCH_OSF=m
+CONFIG_RPS=y
+
+CONFIG_SCSI_MOD=y
+
+
-
+CONFIG_VGA_ARB_MAX_GPUS=2
+CONFIG_USB_EHCI_TT_NEWSCHED=y
-
-CONFIG_INOTIFY=y
+CONFIG_PAX_ELFRELOCS=y
+CONFIG_DEFAULT_SECURITY_DAC=y
+CONFIG_DEFAULT_SECURITY=""
-CONFIG_CRYPTO_FIPS=y
-CONFIG_CRYPTO_AEAD2=y
-CONFIG_CRYPTO_BLKCIPHER=m
-CONFIG_CRYPTO_BLKCIPHER2=y
-CONFIG_CRYPTO_RNG=m
-CONFIG_CRYPTO_RNG2=y
-CONFIG_CRYPTO_PCOMP=y
-CONFIG_CRYPTO_MANAGER=y
-CONFIG_CRYPTO_MANAGER2=y
-CONFIG_CRYPTO_WORKQUEUE=y
-CONFIG_CRYPTO_ECB=m
-CONFIG_CRYPTO_CRC32C_INTEL=m
-CONFIG_CRYPTO_MD5=m
-CONFIG_CRYPTO_SHA512=m
+CONFIG_CRYPTO_AES_586=m
-CONFIG_CRYPTO_DES=m
-CONFIG_CRYPTO_ZLIB=y
-CONFIG_CRYPTO_ANSI_CPRNG=m
-CONFIG_ZLIB_INFLATE=y
-CONFIG_ZLIB_DEFLATE=y

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 26 Dec 2010 at 1:59, Michael Orlitzky wrote:

> I've got (at least) two servers that lose their root partition after
> this upgrade. One of them has an HP cciss SCSI RAID controller; the
> other has a single IDE hard drive. Assuming the problem is something
> common, I'll stick to describing the one with the array for now.

which grsec is this ebuild based on? my guess is that it's a recent PaX/UDEREF
hardening that's causing this and should be mostly fixed now except for the
IP checksum code fix which i'll release soon. in the meantime you can disable
UDEREF. if you don't have it enabled then i don't know what it is, we'll need
more debugging, let me know.

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 12/26/2010 03:46 AM, pageexec@freemail.hu wrote:
> On 26 Dec 2010 at 1:59, Michael Orlitzky wrote:
>
>> I've got (at least) two servers that lose their root partition after
>> this upgrade. One of them has an HP cciss SCSI RAID controller; the
>> other has a single IDE hard drive. Assuming the problem is something
>> common, I'll stick to describing the one with the array for now.
>
> which grsec is this ebuild based on? my guess is that it's a recent PaX/UDEREF
> hardening that's causing this and should be mostly fixed now except for the
> IP checksum code fix which i'll release soon. in the meantime you can disable
> UDEREF. if you don't have it enabled then i don't know what it is, we'll need
> more debugging, let me know.

The hardened-patches contains the following:

4423_grsec-remove-protected-paths.patch
4420_grsecurity-2.2.1-2.6.36.2-201012121726.patch
4435_grsec-kconfig-gentoo.patch
4421_grsec-remove-localversion-grsec.patch
4425_grsec-pax-without-grsec.patch
4430_grsec-kconfig-default-gids.patch
4422_grsec-mute-warnings.patch

I do have UDEREF enabled:

# grep UDEREF .config
CONFIG_PAX_MEMORY_UDEREF=y

I can try disabling it when I'd be willing to drive to work and reboot
the thing.

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 12/26/2010 03:46 AM, pageexec@freemail.hu wrote:
> On 26 Dec 2010 at 1:59, Michael Orlitzky wrote:
>
>> I've got (at least) two servers that lose their root partition after
>> this upgrade. One of them has an HP cciss SCSI RAID controller; the
>> other has a single IDE hard drive. Assuming the problem is something
>> common, I'll stick to describing the one with the array for now.
>
> which grsec is this ebuild based on? my guess is that it's a recent PaX/UDEREF
> hardening that's causing this and should be mostly fixed now except for the
> IP checksum code fix which i'll release soon. in the meantime you can disable
> UDEREF. if you don't have it enabled then i don't know what it is, we'll need
> more debugging, let me know.
>

I'll repeat what I said in the bug report here
(See https://bugs.gentoo.org/show_bug.cgi?id=349705)

hardened-sources-2.6.32-r31 has grsecurity-2.2.1-2.6.32.27-201012121726

hardened-sources-2.6.36-r6 has grsecurity-2.2.1-2.6.36.2-201012121726


What's even stranger is that I have six HP Proliant DL 385 G7, all with
the following (partial) fstab:

/dev/cciss/c0d0p1 /boot ext2 noauto,noatime 1 2
/dev/cciss/c0d0p3 / ext4 noatime 0 1
/dev/cciss/c0d0p2 none swap sw 0 0

None of which showed a panic.


--
Anthony G. Basile, Ph.D.
Gentoo Developer

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 26 Dec 2010 at 12:06, Michael Orlitzky wrote:

> I do have UDEREF enabled:
>
> # grep UDEREF .config
> CONFIG_PAX_MEMORY_UDEREF=y
>
> I can try disabling it when I'd be willing to drive to work and reboot
> the thing.

ok, in this case don't worry about it as i'm sure it's a known bug.
if the next grsec patch (after 12.22) still fails on you then do let
us know though.

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 12/26/2010 03:46 AM, pageexec@freemail.hu wrote:
> On 26 Dec 2010 at 1:59, Michael Orlitzky wrote:
>
>> I've got (at least) two servers that lose their root partition after
>> this upgrade. One of them has an HP cciss SCSI RAID controller; the
>> other has a single IDE hard drive. Assuming the problem is something
>> common, I'll stick to describing the one with the array for now.
>
> which grsec is this ebuild based on? my guess is that it's a recent PaX/UDEREF
> hardening that's causing this and should be mostly fixed now except for the
> IP checksum code fix which i'll release soon. in the meantime you can disable
> UDEREF. if you don't have it enabled then i don't know what it is, we'll need
> more debugging, let me know.
>

Within 24 hours I'll have the following ebuilds on the tree marked ~arch:

hardened-sources-2.6.32-r32
hardened-sources-2.6.36-r7

They are based on the very latest grsec patches. Can users who hit the
panic test them?

--
Anthony G. Basile, Ph.D.
Gentoo Developer

Disappearing root on 2.6.36-hardened-r6 upgrade
 

I don't know if it is related or not. I don't use ext4 and have no
symptoms of disappearing root. I attach a photo taken using a recent
kernel. The latest crashes I've experienced for the past few months
prevented syncing, so didn't get logged. The other screen capture is
older, may not be relevant nowdays.

Thx:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2010.December 26.(V) 09:46 idÅ‘pontban pageexec@freemail.hu ezt Ã*rta:
> On 26 Dec 2010 at 1:59, Michael Orlitzky wrote:
>
>> I've got (at least) two servers that lose their root partition after
>> this upgrade. One of them has an HP cciss SCSI RAID controller; the
>> other has a single IDE hard drive. Assuming the problem is something
>> common, I'll stick to describing the one with the array for now.
>
> which grsec is this ebuild based on? my guess is that it's a recent
> PaX/UDEREF
> hardening that's causing this and should be mostly fixed now except for
> the
> IP checksum code fix which i'll release soon. in the meantime you can
> disable
> UDEREF. if you don't have it enabled then i don't know what it is, we'll
> need
> more debugging, let me know.
>
>

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 12/26/2010 12:57 PM, pageexec@freemail.hu wrote:
> On 26 Dec 2010 at 12:06, Michael Orlitzky wrote:
>
>> I do have UDEREF enabled:
>>
>> # grep UDEREF .config
>> CONFIG_PAX_MEMORY_UDEREF=y
>>
>> I can try disabling it when I'd be willing to drive to work and reboot
>> the thing.
>
> ok, in this case don't worry about it as i'm sure it's a known bug.
> if the next grsec patch (after 12.22) still fails on you then do let
> us know though.

Challenge accepted. I'm dressed, the car's cleaned off, and I'm
recompiling with UDEREF=n.

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 12/26/2010 12:57 PM, pageexec@freemail.hu wrote:
> On 26 Dec 2010 at 12:06, Michael Orlitzky wrote:
>
>> I do have UDEREF enabled:
>>
>> # grep UDEREF .config
>> CONFIG_PAX_MEMORY_UDEREF=y
>>
>> I can try disabling it when I'd be willing to drive to work and reboot
>> the thing.
>
> ok, in this case don't worry about it as i'm sure it's a known bug.
> if the next grsec patch (after 12.22) still fails on you then do let
> us know though.

To my mild surprise, the box came back up. Disabling UDEREF fixed it.
I'll give the new ~arch ebuilds a try, too, when they become available.

Disappearing root on 2.6.36-hardened-r6 upgrade
 

On 26 Dec 2010 at 14:09, Michael Orlitzky wrote:

> Challenge accepted. I'm dressed, the car's cleaned off, and I'm
> recompiling with UDEREF=n.

passing pax_nouderef on the kernel cmdline should be enough ;)