Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo Development (http://www.linux-archive.org/gentoo-development/)
-   -   Debugging a Linux kernel deadlock when doing swap on ZFS (http://www.linux-archive.org/gentoo-development/655477-debugging-linux-kernel-deadlock-when-doing-swap-zfs.html)

Richard Yao 04-12-2012 11:30 PM

Debugging a Linux kernel deadlock when doing swap on ZFS
 
I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
when it tries to use it. I am having trouble getting a backtrace.

Does anyone have any suggestions on how I could debug this?

Mike Frysinger 04-13-2012 12:44 AM

Debugging a Linux kernel deadlock when doing swap on ZFS
 
On Thursday 12 April 2012 19:30:08 Richard Yao wrote:
> I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
> and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
> when it tries to use it. I am having trouble getting a backtrace.
>
> Does anyone have any suggestions on how I could debug this?

networked swap has always been hairy. your best bet would be to ask the zfs
guys first. otherwise, g'luck -- the mm in the kernel is bad enough without
adding in the network and filesystem stacks.

otherwise, there's quick options like enabling lockdep checking in the kernel.
or the NMI watchdog.
-mike

Sergei Trofimovich 04-13-2012 05:27 AM

Debugging a Linux kernel deadlock when doing swap on ZFS
 
On Thu, 12 Apr 2012 19:30:08 -0400
Richard Yao <ryao@cs.stonybrook.edu> wrote:

> I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
> and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
> when it tries to use it. I am having trouble getting a backtrace.
>
> Does anyone have any suggestions on how I could debug this?
>
If it's that easy to reproduce, I'd suggest to try it in usermode
linux (ARCH=um make). It's a linux kernel as unprivileged userspace
process. Easy to attach with gdb, add printk and etc.
Does not support SMP.

- swap on FS has some optimizations which are not very well suited
to copy-on-write files. kernel picks file block map from filesystem
and tries to use it directly (code might be shared with FIBMAP
implementation). It's a tricky part to handle.
- triple-check all ZFS memory allocations go at least via GFP_NOFS
allocations.

--

Sergei

Richard Yao 04-15-2012 09:48 AM

Debugging a Linux kernel deadlock when doing swap on ZFS
 
Thanks everyone for your extremely useful tips. I seem to have it
working now. The problem is that ZFS does memory allocations when asked
to write things.

A makeshift solution is to do `echo 524288 >
/proc/sys/vm/min_free_kbytes`. A more permanent fix will take more time
to produce, but at least I have a handle on the situation now.


All times are GMT. The time now is 01:05 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.