Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

One the last rounds of testing the new release of OLPCXS, I rebuilt it
with fresh packages from F9 update and started testing the installer.
Funny thing, the install did not complete -- instead, the machine
would switch off after spending a few minutes trying to install
selinux-policy-targeted.

After a few attempts to diagnose the problem, I managed to see vmstat
go all the way down to almost 0 memory just before the machine turned
itself off. This particular machine has ~980MB RAM available to the
OS. Tested on another machine with a proper 1GB memory, vmstat hit
bottom at ~10MB free while installing selinux-policy-targeted but
quickly recovered.

I know I've installed earlier F9 based spins on machines with 512 MB
of physical RAM so this seems like a fairly bad regression, specially
considering that I'll soon need to install this on a machine with
256MB RAM.

So I suspect we have 2 problems

- Selinux-policy-targeted instalation seems to have balooned into a
memory hog between f9-release and f9-updates

- Anaconda OOMs without a warning or useful message to the user

Has anyone else seen this? Diagnostics to recommend? I can
successfully log anything to disk until moments before the OOM
shutdown.

Will post version numbers of the selinux rpm tomorrow when I get back
in front of the offending machine and installer image. Apologies for
the vagueness :-)

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

> - Anaconda OOMs without a warning or useful message to the user

I don't believe there's any way for anaconda to know this, and there's
certainly no way for us to do anything about it.

- Chris

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

On Fri, Nov 7, 2008 at 9:49 AM, Daniel J Walsh <dwalsh@redhat.com> wrote:
> selinux-policy-targeted is a memory hog, but it should not have changed
> that drastically in updates.
>
> Is this repeatable?

100% repeatable on the 3 attempts on the lower-mem machine. On the same machin

The package selinux-policy-targeted 3.3.1 103.fc9 -- I am running a
couple of additional installs to gather more information.

On Fri, Nov 7, 2008 at 9:59 AM, Chris Lumens <clumens@redhat.com> wrote:
>> - Anaconda OOMs without a warning or useful message to the user
>
> I don't believe there's any way for anaconda to know this, and there's
> certainly no way for us to do anything about it.

Well, the behaviour is really weird. Perhaps there is no OOM killer in
place during an anaconda install? If there was, it'd expect the rpm
process or anaconda to be shot down -- but the machine is halting
instead, it literally switches off.

Whatever is bootstrapping anaconda (init script in the initrd?) should
be able to at least see the odd exit status and echo a "something went
wrong" msg...? Maybe that's the problem?

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

On Fri, Nov 7, 2008 at 9:49 AM, Daniel J Walsh <dwalsh@redhat.com> wrote:
> selinux-policy-targeted is a memory hog, but it should not have changed
> that drastically in updates.
>
> Is this repeatable?

100% repeatable on the 3 attempts on the lower-mem machine. On the same machin

The package selinux-policy-targeted 3.3.1 103.fc9 -- I am running a
couple of additional installs to gather more information.

On Fri, Nov 7, 2008 at 9:59 AM, Chris Lumens <clumens@redhat.com> wrote:
>> - Anaconda OOMs without a warning or useful message to the user
>
> I don't believe there's any way for anaconda to know this, and there's
> certainly no way for us to do anything about it.

Well, the behaviour is really weird. Perhaps there is no OOM killer in
place during an anaconda install? If there was, it'd expect the rpm
process or anaconda to be shot down -- but the machine is halting
instead, it literally switches off.

Whatever is bootstrapping anaconda (init script in the initrd?) should
be able to at least see the odd exit status and echo a "something went
wrong" msg...? Maybe that's the problem?

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

On Fri, Nov 7, 2008 at 10:38 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Fri, Nov 7, 2008 at 9:49 AM, Daniel J Walsh <dwalsh@redhat.com> wrote:
>> selinux-policy-targeted is a memory hog, but it should not have changed
>> that drastically in updates.
>>
>> Is this repeatable?
>
> 100% repeatable on the 3 attempts on the lower-mem machine. On the same machin

Ok - now I have a few more tests

- it does not happen in text mode
- it does not happen in gui mode if I'm looking at the VT
- it does happen in gui mode if I am looking at X

And according to the logs, it's does not appear to be an OOM -- the
last reading shows 32MB free. Or it's such a sudden OOM that it never
gets written to disk. Memory usage logs are captured doing (in the
VT):

tail -f /mnt/sysimage/root/install.log >> /mnt/sysimage/root/instmem.log &
vmstat 1 >> /mnt/sysimage/root/instmem.log &

so I have a log showing mem usage during each rpm installation. The
memory usage is not specially high.

Perhaps something else is happening during that rpm installation. Is
there any command that I can pass to anaconda to write a more verbose
log?

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

On Fri, Nov 7, 2008 at 10:38 AM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Fri, Nov 7, 2008 at 9:49 AM, Daniel J Walsh <dwalsh@redhat.com> wrote:
>> selinux-policy-targeted is a memory hog, but it should not have changed
>> that drastically in updates.
>>
>> Is this repeatable?
>
> 100% repeatable on the 3 attempts on the lower-mem machine. On the same machin

Ok - now I have a few more tests

- it does not happen in text mode
- it does not happen in gui mode if I'm looking at the VT
- it does happen in gui mode if I am looking at X

And according to the logs, it's does not appear to be an OOM -- the
last reading shows 32MB free. Or it's such a sudden OOM that it never
gets written to disk. Memory usage logs are captured doing (in the
VT):

tail -f /mnt/sysimage/root/install.log >> /mnt/sysimage/root/instmem.log &
vmstat 1 >> /mnt/sysimage/root/instmem.log &

so I have a log showing mem usage during each rpm installation. The
memory usage is not specially high.

Perhaps something else is happening during that rpm installation. Is
there any command that I can pass to anaconda to write a more verbose
log?

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Anaconda installs OOM with selinux-policy-targeted rpm from F9 updates
 

On Fri, Nov 07, 2008 at 01:13:38AM -0500, Martin Langhoff wrote:
> One the last rounds of testing the new release of OLPCXS, I rebuilt it
> with fresh packages from F9 update and started testing the installer.
> Funny thing, the install did not complete -- instead, the machine
> would switch off after spending a few minutes trying to install
> selinux-policy-targeted.
>
> After a few attempts to diagnose the problem, I managed to see vmstat
> go all the way down to almost 0 memory just before the machine turned
> itself off. This particular machine has ~980MB RAM available to the
> OS. Tested on another machine with a proper 1GB memory, vmstat hit
> bottom at ~10MB free while installing selinux-policy-targeted but
> quickly recovered.
>
> I know I've installed earlier F9 based spins on machines with 512 MB
> of physical RAM so this seems like a fairly bad regression, specially
> considering that I'll soon need to install this on a machine with
> 256MB RAM.
>
> So I suspect we have 2 problems
>
> - Selinux-policy-targeted instalation seems to have balooned into a
> memory hog between f9-release and f9-updates
>
> - Anaconda OOMs without a warning or useful message to the user
>
> Has anyone else seen this? Diagnostics to recommend? I can
> successfully log anything to disk until moments before the OOM
> shutdown.

Did you configure any swap ?
If no, then there's really not much that we can do if something
uses more than available system RAM.

If you did, it might be interesting to try (from tty2)
echo 1 > /proc/sys/vm/would_have_oomkilled

That will prevent the actually 'killing', but will still
log all the same output that the oomkiller would have spewed.
It might be interesting to see that output.

If you're drastically low enough on memory to invoke an OOM kill
though, setting that sysctl may just mean the system livelocks.

Dave

--
http://www.codemonkey.org.uk

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list