2.6.37 GFS/CLVM/DLM trouble II

03-20-2011, 06:01 PM

Hello Stephen et al,

some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
dumps, I weren't able to reproduce till now.
Today, the on my testing machine, GFS got stuck again. I also noticed
that clustered LVM is also stuck on it, so I guess the problem is
somewhere in the DLM code, not GFS.

Here are kernel backtraces:

[182189.107631] INFO: task clvmd:17723 blocked for more than 120
seconds.
[182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[182189.107634] clvmd D ffffffff8140a4c0 0 17723 1
0x00000000
[182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000
00000000000116c0
[182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530
ffff88013fcd0000
[182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8
ffff8800853c1c98
[182189.107647] Call Trace:
[182189.107651] [<ffffffff810d5025>] ?
get_page_from_freelist+0x3b5/0x510
[182189.107654] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107656] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107659] [<ffffffff8136d205>]
rwsem_down_failed_common+0xb5/0x130
[182189.107663] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
[182189.107665] [<ffffffff811d4c44>]
call_rwsem_down_read_failed+0x14/0x30
[182189.107668] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
[182189.107673] [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
[182189.107676] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107679] [<ffffffff8110e23e>] ?
kmem_cache_alloc_notrace+0x9e/0xc0
[182189.107684] [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
[182189.107687] [<ffffffff811a9cde>] ?
security_file_permission+0x1e/0x90
[182189.107689] [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
[182189.107691] [<ffffffff8111aef9>] vfs_write+0xc9/0x190
[182189.107694] [<ffffffff8111b640>] sys_write+0x50/0x90
[182189.107697] [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
[182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
seconds.
[182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2
0x00000000
[182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0
00000000000116c0
[182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20
ffff88013fcd5c40
[182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
ffff88013fcd0000
[182189.107720] Call Trace:
[182189.107723] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107726] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107729] [<ffffffff8136d205>]
rwsem_down_failed_common+0xb5/0x130
[182189.107731] [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
[182189.107734] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
[182189.107737] [<ffffffff811d4c44>]
call_rwsem_down_read_failed+0x14/0x30
[182189.107740] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
[182189.107745] [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
[182189.107747] [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
[182189.107750] [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
[182189.107760] [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
[182189.107769] [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
[182189.107777] [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
[182189.107783] [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
[182189.107789] [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
[182189.107796] [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
[182189.107804] [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
[182189.107812] [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
[gfs2]
[182189.107815] [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
[182189.107823] [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
[gfs2]
[182189.107831] [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
[182189.107834] [<ffffffff8106b130>] ?
autoremove_wake_function+0x0/0x40
[182189.107837] [<ffffffff8136d77d>] ?
_raw_spin_unlock_irqrestore+0x1d/0x50
[182189.107846] [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
[182189.107848] [<ffffffff8106ac06>] kthread+0x96/0xa0
[182189.107851] [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
[182189.107854] [<ffffffff8106ab70>] ? kthread+0x0/0xa0
[182189.107857] [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10

and here debugfs DLM lock dumps:

[root@vbox5 pcmk:lvs]# cat glocks
G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
[gfs2]
G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
[gfs2]
H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]

The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
GFS is handled by corosync/pacemaker cluster.
Could somebody please help me to debug it? I can keep the machine in
hung state for some time as it's testing box...

Thanks a lot in advance!

with best regards

nik

--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

03-21-2011, 08:50 AM

Hi,

On Sun, 2011-03-20 at 20:01 +0100, Nikola Ciprich wrote:
> Hello Stephen et al,
>
> some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
> dumps, I weren't able to reproduce till now.
> Today, the on my testing machine, GFS got stuck again. I also noticed
> that clustered LVM is also stuck on it, so I guess the problem is
> somewhere in the DLM code, not GFS.
>
> Here are kernel backtraces:
>
> [182189.107631] INFO: task clvmd:17723 blocked for more than 120
> seconds.
> [182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107634] clvmd D ffffffff8140a4c0 0 17723 1
> 0x00000000
> [182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000
> 00000000000116c0
> [182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530
> ffff88013fcd0000
> [182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8
> ffff8800853c1c98
> [182189.107647] Call Trace:
> [182189.107651] [<ffffffff810d5025>] ?
> get_page_from_freelist+0x3b5/0x510
> [182189.107654] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107656] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107659] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107663] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107665] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107668] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107673] [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
> [182189.107676] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107679] [<ffffffff8110e23e>] ?
> kmem_cache_alloc_notrace+0x9e/0xc0
> [182189.107684] [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
> [182189.107687] [<ffffffff811a9cde>] ?
> security_file_permission+0x1e/0x90
> [182189.107689] [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
> [182189.107691] [<ffffffff8111aef9>] vfs_write+0xc9/0x190
> [182189.107694] [<ffffffff8111b640>] sys_write+0x50/0x90
> [182189.107697] [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
> [182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
> seconds.
> [182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2
> 0x00000000
> [182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0
> 00000000000116c0
> [182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20
> ffff88013fcd5c40
> [182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
> ffff88013fcd0000
> [182189.107720] Call Trace:
> [182189.107723] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107726] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107729] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107731] [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
> [182189.107734] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107737] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107740] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107745] [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
> [182189.107747] [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
> [182189.107750] [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
> [182189.107760] [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
> [182189.107769] [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
> [182189.107777] [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
> [182189.107783] [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
> [182189.107789] [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
> [182189.107796] [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
> [182189.107804] [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
> [182189.107812] [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> [182189.107815] [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
> [182189.107823] [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
> [gfs2]
> [182189.107831] [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
> [182189.107834] [<ffffffff8106b130>] ?
> autoremove_wake_function+0x0/0x40
> [182189.107837] [<ffffffff8136d77d>] ?
> _raw_spin_unlock_irqrestore+0x1d/0x50
> [182189.107846] [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
> [182189.107848] [<ffffffff8106ac06>] kthread+0x96/0xa0
> [182189.107851] [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
> [182189.107854] [<ffffffff8106ab70>] ? kthread+0x0/0xa0
> [182189.107857] [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10
>
So there are two processes, both waiting on an rwsem which is somewhere
in dlm.

> and here debugfs DLM lock dumps:
>
This is a glock dump not a dlm lock dump.

> [root@vbox5 pcmk:lvs]# cat glocks
> G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
> G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
> G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
> G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
> G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
> G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
> H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
> G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
> H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
> H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
> [gfs2]
> H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
> I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
> G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
> H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
> I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
> G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
> G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
> I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
> G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
> G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
> G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
> G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
> I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
> G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
> G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
> H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
> I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
> G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
> H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
> G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
> G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
>
> The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
> GFS is handled by corosync/pacemaker cluster.
> Could somebody please help me to debug it? I can keep the machine in
> hung state for some time as it's testing box...
>
> Thanks a lot in advance!
>
> with best regards
>
> nik
>
>
Do you have any log messages relating to recovery? I'm wondering if that
might have failed and be the reason for these messages. It would be
useful to have a dump from gfs_control for example,

Steve.

03-21-2011, 03:47 PM

Hi,

On Mon, 2011-03-21 at 17:08 +0100, Nikola Ciprich wrote:
> > This is a glock dump not a dlm lock dump.
> ouch, I see where is the problem, I don't have DLM debugging enabled in my kernel

> So I'll enable it and wait till I can reproduce the problem again

>
> > Do you have any log messages relating to recovery? I'm wondering if that
> > might have failed and be the reason for these messages. It would be
> > useful to have a dump from gfs_control for example,
>
> I don't see anything in the logs..
>
> I'm not sure whether I'm trying to use gfs2_tool lockdump correctly (I add GFS2
> mountpoint as parameter), but it gets stuck too...
>
>
> Thanks for Your time and sorry to bother..
> have a nice day
> n.
>
It is probably easier to get the information directly
from /sys/kernel/debug/[gfs2|dlm]/* than to use the tool. You need to
have debugfs mounted there (I assume you have since you were able to get
the gfs2 lockdump above)

Steve.