FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Cluster Development

 
 
LinkBack Thread Tools
 
Old 03-20-2011, 06:01 PM
Nikola Ciprich
 
Default 2.6.37 GFS/CLVM/DLM trouble II

Hello Stephen et al,

some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
dumps, I weren't able to reproduce till now.
Today, the on my testing machine, GFS got stuck again. I also noticed
that clustered LVM is also stuck on it, so I guess the problem is
somewhere in the DLM code, not GFS.

Here are kernel backtraces:

[182189.107631] INFO: task clvmd:17723 blocked for more than 120
seconds.
[182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[182189.107634] clvmd D ffffffff8140a4c0 0 17723 1
0x00000000
[182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000
00000000000116c0
[182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530
ffff88013fcd0000
[182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8
ffff8800853c1c98
[182189.107647] Call Trace:
[182189.107651] [<ffffffff810d5025>] ?
get_page_from_freelist+0x3b5/0x510
[182189.107654] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107656] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107659] [<ffffffff8136d205>]
rwsem_down_failed_common+0xb5/0x130
[182189.107663] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
[182189.107665] [<ffffffff811d4c44>]
call_rwsem_down_read_failed+0x14/0x30
[182189.107668] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
[182189.107673] [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
[182189.107676] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107679] [<ffffffff8110e23e>] ?
kmem_cache_alloc_notrace+0x9e/0xc0
[182189.107684] [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
[182189.107687] [<ffffffff811a9cde>] ?
security_file_permission+0x1e/0x90
[182189.107689] [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
[182189.107691] [<ffffffff8111aef9>] vfs_write+0xc9/0x190
[182189.107694] [<ffffffff8111b640>] sys_write+0x50/0x90
[182189.107697] [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
[182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
seconds.
[182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2
0x00000000
[182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0
00000000000116c0
[182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20
ffff88013fcd5c40
[182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
ffff88013fcd0000
[182189.107720] Call Trace:
[182189.107723] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107726] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
[182189.107729] [<ffffffff8136d205>]
rwsem_down_failed_common+0xb5/0x130
[182189.107731] [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
[182189.107734] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
[182189.107737] [<ffffffff811d4c44>]
call_rwsem_down_read_failed+0x14/0x30
[182189.107740] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
[182189.107745] [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
[182189.107747] [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
[182189.107750] [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
[182189.107760] [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
[182189.107769] [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
[182189.107777] [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
[182189.107783] [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
[182189.107789] [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
[182189.107796] [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
[182189.107804] [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
[182189.107812] [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
[gfs2]
[182189.107815] [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
[182189.107823] [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
[gfs2]
[182189.107831] [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
[182189.107834] [<ffffffff8106b130>] ?
autoremove_wake_function+0x0/0x40
[182189.107837] [<ffffffff8136d77d>] ?
_raw_spin_unlock_irqrestore+0x1d/0x50
[182189.107846] [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
[182189.107848] [<ffffffff8106ac06>] kthread+0x96/0xa0
[182189.107851] [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
[182189.107854] [<ffffffff8106ab70>] ? kthread+0x0/0xa0
[182189.107857] [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10

and here debugfs DLM lock dumps:

[root@vbox5 pcmk:lvs]# cat glocks
G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
[gfs2]
G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
[gfs2]
H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]

The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
GFS is handled by corosync/pacemaker cluster.
Could somebody please help me to debug it? I can keep the machine in
hung state for some time as it's testing box...

Thanks a lot in advance!

with best regards

nik


--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------
 
Old 03-21-2011, 08:50 AM
Steven Whitehouse
 
Default 2.6.37 GFS/CLVM/DLM trouble II

Hi,

On Sun, 2011-03-20 at 20:01 +0100, Nikola Ciprich wrote:
> Hello Stephen et al,
>
> some time ago, I reported GFS2 hangs. You asked me to obtain DLM lock
> dumps, I weren't able to reproduce till now.
> Today, the on my testing machine, GFS got stuck again. I also noticed
> that clustered LVM is also stuck on it, so I guess the problem is
> somewhere in the DLM code, not GFS.
>
> Here are kernel backtraces:
>
> [182189.107631] INFO: task clvmd:17723 blocked for more than 120
> seconds.
> [182189.107633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107634] clvmd D ffffffff8140a4c0 0 17723 1
> 0x00000000
> [182189.107637] ffff8800853c1ca0 0000000000000086 0000000000000000
> 00000000000116c0
> [182189.107641] ffff88013b7348d8 0000000000000001 ffff88013b734530
> ffff88013fcd0000
> [182189.107644] ffff8800853c1fd8 0000000000000001 0000000001c225b8
> ffff8800853c1c98
> [182189.107647] Call Trace:
> [182189.107651] [<ffffffff810d5025>] ?
> get_page_from_freelist+0x3b5/0x510
> [182189.107654] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107656] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107659] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107663] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107665] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107668] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107673] [<ffffffffa0548aa2>] dlm_user_request+0x42/0x260 [dlm]
> [182189.107676] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107679] [<ffffffff8110e23e>] ?
> kmem_cache_alloc_notrace+0x9e/0xc0
> [182189.107684] [<ffffffffa0551b04>] device_write+0x684/0x880 [dlm]
> [182189.107687] [<ffffffff811a9cde>] ?
> security_file_permission+0x1e/0x90
> [182189.107689] [<ffffffff8111a894>] ? rw_verify_area+0x74/0xf0
> [182189.107691] [<ffffffff8111aef9>] vfs_write+0xc9/0x190
> [182189.107694] [<ffffffff8111b640>] sys_write+0x50/0x90
> [182189.107697] [<ffffffff810024fb>] system_call_fastpath+0x16/0x1b
> [182189.107705] INFO: task gfs2_quotad:22599 blocked for more than 120
> seconds.
> [182189.107706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [182189.107707] gfs2_quotad D ffffffff8140a4c0 0 22599 2
> 0x00000000
> [182189.107711] ffff880113d0ba88 0000000000000046 00000000000116c0
> 00000000000116c0
> [182189.107714] ffff8801141bb1c8 0000000000000002 ffff8801141bae20
> ffff88013fcd5c40
> [182189.107717] ffff880113d0bfd8 ffff880113d0b9b0 0000000081046cd4
> ffff88013fcd0000
> [182189.107720] Call Trace:
> [182189.107723] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107726] [<ffffffff8103aee1>] ? get_parent_ip+0x11/0x50
> [182189.107729] [<ffffffff8136d205>]
> rwsem_down_failed_common+0xb5/0x130
> [182189.107731] [<ffffffff81035fb1>] ? cpuacct_charge+0x61/0x70
> [182189.107734] [<ffffffff8136d2b5>] rwsem_down_read_failed+0x15/0x17
> [182189.107737] [<ffffffff811d4c44>]
> call_rwsem_down_read_failed+0x14/0x30
> [182189.107740] [<ffffffff8136c65d>] ? down_read+0x2d/0x40
> [182189.107745] [<ffffffffa0547039>] dlm_lock+0x59/0x180 [dlm]
> [182189.107747] [<ffffffff81045ae2>] ? update_curr+0xb2/0x170
> [182189.107750] [<ffffffff810374df>] ? hrtick_update+0x2f/0x40
> [182189.107760] [<ffffffffa05885d3>] gdlm_lock+0xd3/0x120 [gfs2]
> [182189.107769] [<ffffffffa05887f0>] ? gdlm_ast+0x0/0x160 [gfs2]
> [182189.107777] [<ffffffffa0588620>] ? gdlm_bast+0x0/0x50 [gfs2]
> [182189.107783] [<ffffffffa056a62c>] do_xmote+0x18c/0x280 [gfs2]
> [182189.107789] [<ffffffffa056a7b1>] run_queue+0x91/0x260 [gfs2]
> [182189.107796] [<ffffffffa056aac3>] gfs2_glock_nq+0xc3/0x3a0 [gfs2]
> [182189.107804] [<ffffffffa0584f49>] gfs2_statfs_sync+0x59/0x1a0 [gfs2]
> [182189.107812] [<ffffffffa0584f41>] ? gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> [182189.107815] [<ffffffff8103c64d>] ? sub_preempt_count+0x9d/0xd0
> [182189.107823] [<ffffffffa057dbf7>] quotad_check_timeo+0x57/0x90
> [gfs2]
> [182189.107831] [<ffffffffa057f637>] gfs2_quotad+0x207/0x240 [gfs2]
> [182189.107834] [<ffffffff8106b130>] ?
> autoremove_wake_function+0x0/0x40
> [182189.107837] [<ffffffff8136d77d>] ?
> _raw_spin_unlock_irqrestore+0x1d/0x50
> [182189.107846] [<ffffffffa057f430>] ? gfs2_quotad+0x0/0x240 [gfs2]
> [182189.107848] [<ffffffff8106ac06>] kthread+0x96/0xa0
> [182189.107851] [<ffffffff810032d4>] kernel_thread_helper+0x4/0x10
> [182189.107854] [<ffffffff8106ab70>] ? kthread+0x0/0xa0
> [182189.107857] [<ffffffff810032d0>] ? kernel_thread_helper+0x0/0x10
>
So there are two processes, both waiting on an rwsem which is somewhere
in dlm.

> and here debugfs DLM lock dumps:
>
This is a glock dump not a dlm lock dump.

> [root@vbox5 pcmk:lvs]# cat glocks
> G: s:EX n:2/20188 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:22/131464 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d24 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/154916 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/102b7 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:101/196988 t:8 f:0x00 d:0x00000000 s:941
> G: s:SH n:5/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20185 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:19/131461 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20189 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/18 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:3/24 t:4 f:0x00 d:0x00000201 s:3864
> G: s:UN n:2/25d0c f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:8/131450 t:8 f:0x00 d:0x00000000 s:1822
> G: s:EX n:2/2018b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:25/131467 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/3017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:75/196987 t:8 f:0x00 d:0x00000000 s:1170
> G: s:SH n:5/3017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20180 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d32 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20184 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:18/131460 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/2018b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2018a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:24/131466 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/10839 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:1/2 f:Iq t:SH d:EX/0 a:0 r:3
> G: s:UN n:2/102ab f:lIq t:EX d:EX/0 a:0 r:4
> H: s:EX f:cW e:0 p:22599 [gfs2_quotad] gfs2_statfs_sync+0x51/0x1a0
> [gfs2]
> G: s:EX n:2/1053a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/66874 t:8 f:0x00 d:0x00000000 s:3126995
> G: s:SH n:5/25d30 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d30 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:37/154928 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d38 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ab f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d38 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:45/154936 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d26 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:27/154918 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/16 f:Iq t:SH d:EX/0 a:0 r:6
> H: s:SH f:H e:0 p:17736 [mc] gfs2_lookupi+0xbc/0x1c0 [gfs2]
> H: s:EX f:W e:0 p:17711 [flush-253:6] gfs2_write_inode+0x7a/0x170
> [gfs2]
> H: s:SH f:AW e:0 p:18238 [ls] gfs2_getattr+0x89/0xf0 [gfs2]
> I: n:1/22 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/20180 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:14/131456 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:4/0 f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/3017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017b f:yIq t:EX d:EX/0 a:1 r:3
> I: n:9/131451 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/25d24 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d3a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:47/154938 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/10839 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/67641 t:4 f:0x00 d:0x00000001 s:3864
> G: s:EX n:2/25d11 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:6/154897 t:8 f:0x00 d:0x00000000 s:392
> G: s:SH n:2/19 f:Iq t:SH d:EX/0 a:0 r:4
> H: s:SH f:eEcH e:0 p:22575 [(ended)] init_journal+0x63f/0x9d0 [gfs2]
> I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
> G: s:EX n:2/25d10 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:5/154896 t:8 f:0x00 d:0x00000000 s:1423
> G: s:SH n:5/17 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d10 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1083a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/2017c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:10/131452 t:8 f:0x00 d:0x00000000 s:1621
> G: s:SH n:5/20186 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2c f:yIq t:EX d:EX/0 a:1 r:3
> I: n:33/154924 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d2e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d0f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:SH n:5/2017c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102b9 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2018a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/102ac f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/102ac f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/20185 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/1083a f:Iq t:SH d:EX/0 a:0 r:3
> I: n:5/67642 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/805b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:2/805b f:Iq t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/25d0e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:103/196989 t:8 f:0x00 d:0x00000000 s:1065
> G: s:EX n:2/20187 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:21/131463 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20186 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:20/131462 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/3017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/2017b f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d2a f:yIq t:EX d:EX/0 a:1 r:3
> I: n:31/154922 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/1053a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:4613 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20181 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:15/131457 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d2e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:35/154926 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/30179 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:73/196985 t:8 f:0x00 d:0x00000000 s:1084
> G: s:UN n:2/102b7 f: t:UN d:EX/0 a:0 r:2
> G: s:EX n:2/2017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:13/131455 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/102b8 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:1/66232 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:1/1 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:eEH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:EX n:2/2017d f:yIq t:EX d:EX/0 a:1 r:3
> I: n:11/131453 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20189 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:23/131465 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d34 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:41/154932 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/25d0f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:4/154895 t:10 f:0x00 d:0x00000000 s:10
> G: s:EX n:2/25d28 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:29/154920 t:8 f:0x00 d:0x00000000 s:1434
> G: s:EX n:2/20182 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:16/131458 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20182 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017f f:yIq t:EX d:EX/0 a:1 r:3
> I: n:85/196991 t:8 f:0x00 d:0x00000000 s:1102
> G: s:SH n:5/2017a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2c f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:1/3 f: t:UN d:EX/0 a:0 r:2
> G: s:SH n:5/2017d f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/20183 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:17/131459 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:2/1009d f:Iq t:SH d:EX/0 a:0 r:2
> G: s:EX n:2/100a0 f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x181/0x250 [gfs2]
> I: n:9/65696 t:8 f:0x00 d:0x00000200 s:1048576
> G: s:EX n:2/25d0e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:3/154894 t:4 f:0x00 d:0x00000001 s:3864
> G: s:SH n:5/2017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d26 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:2/17 f:Iq t:SH d:EX/0 a:0 r:3
> I: n:2/23 t:4 f:0x00 d:0x00000201 s:3864
> G: s:SH n:5/25d11 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22615 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/1009f f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:9/0 f:Iq t:EX d:EX/0 a:0 r:3
> H: s:EX f:eH e:0 p:22575 [(ended)] gfs2_glock_nq_num+0x62/0x90 [gfs2]
> G: s:SH n:5/3017e f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20184 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20187 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/16 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d32 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:39/154930 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/25d34 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/25d36 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:43/154934 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20183 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/1009f f:Iq t:EX d:EX/0 a:0 r:4
> H: s:EX f:H e:0 p:22575 [(ended)] init_per_node+0x14e/0x250 [gfs2]
> I: n:8/65695 t:8 f:0x00 d:0x00000201 s:24
> G: s:SH n:5/30179 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:29119 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d3a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:UN n:5/25d0c f:lq t:SH d:EX/0 a:0 r:4
> H: s:SH f:EW e:0 p:17736 [mc] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/18 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d28 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/100a0 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d36 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/20181 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/102b9 f:yIq t:EX d:EX/0 a:1 r:3
> I: n:2/66233 t:8 f:0x00 d:0x00000000 s:2612512
> G: s:EX n:2/2017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:12/131454 t:8 f:0x00 d:0x00000000 s:1434
> G: s:SH n:5/20188 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:EX n:2/3017e f:yIq t:EX d:EX/0 a:1 r:3
> I: n:84/196990 t:8 f:0x00 d:0x00000000 s:1115
> G: s:SH n:5/19 f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:22575 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
> G: s:SH n:5/25d2a f:Iq t:SH d:EX/0 a:0 r:3
> H: s:SH f:EH e:0 p:23817 [(ended)] gfs2_inode_lookup+0x117/0x250 [gfs2]
>
> The machine is SMP x86_64 running 2.6.37.4 now. DLM, CLVMD as well as
> GFS is handled by corosync/pacemaker cluster.
> Could somebody please help me to debug it? I can keep the machine in
> hung state for some time as it's testing box...
>
> Thanks a lot in advance!
>
> with best regards
>
> nik
>
>
Do you have any log messages relating to recovery? I'm wondering if that
might have failed and be the reason for these messages. It would be
useful to have a dump from gfs_control for example,

Steve.
 
Old 03-21-2011, 03:47 PM
Steven Whitehouse
 
Default 2.6.37 GFS/CLVM/DLM trouble II

Hi,

On Mon, 2011-03-21 at 17:08 +0100, Nikola Ciprich wrote:
> > This is a glock dump not a dlm lock dump.
> ouch, I see where is the problem, I don't have DLM debugging enabled in my kernel
> So I'll enable it and wait till I can reproduce the problem again
>
> > Do you have any log messages relating to recovery? I'm wondering if that
> > might have failed and be the reason for these messages. It would be
> > useful to have a dump from gfs_control for example,
>
> I don't see anything in the logs..
>
> I'm not sure whether I'm trying to use gfs2_tool lockdump correctly (I add GFS2
> mountpoint as parameter), but it gets stuck too...
>
>
> Thanks for Your time and sorry to bother..
> have a nice day
> n.
>
It is probably easier to get the information directly
from /sys/kernel/debug/[gfs2|dlm]/* than to use the tool. You need to
have debugfs mounted there (I assume you have since you were able to get
the gfs2 lockdump above)

Steve.
 

Thread Tools




All times are GMT. The time now is 04:31 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org