[LU-5672] sanity-lfsck test_9b: journal deadlock with lfsck thread

Thu, 2 Oct 2014 19:57:19 +0000

This issue was created by maloo for John Hammond <john.hammond@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/237f334e-4501-11e4-8e4d-5254006e85c2.

The sub-test test_9b failed with the following error:

test failed to respond and timed out

11:31:43:Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) == 17:27:48 (1411666068)
11:31:43:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604
11:31:43:Lustre: *** cfs_fail_loc=1604, val=0***
11:31:43:Lustre: Skipped 4 previous similar messages
11:31:43:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160c
11:31:43:Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r
11:31:43:Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n 		mdd.lustre-MDT0000.lfsck_namespace |
11:31:43:		awk '/^status/ { print $2 }'
11:31:43:INFO: task jbd2/dm-0-8:26472 blocked for more than 120 seconds.
11:31:43:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:31:43:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:31:43:jbd2/dm-0-8   D 0000000000000001     0 26472      2 0x00000080
11:31:43: ffff88007a927d20 0000000000000046 ffff88006ecb36d0 ffff88006ecb36c0
11:31:43: ffff88007a3f8a40 ffff8800023168e8 000000000003c174 ffff880079472080
11:31:43: ffff880079472638 ffff88007a927fd8 000000000000fbc8 ffff880079472638
11:31:43:Call Trace:
11:31:43: [<ffffffff8109b2ce>] ? prepare_to_wait+0x4e/0x80
11:31:43: [<ffffffffa03df80f>] jbd2_journal_commit_transaction+0x19f/0x1500 [jbd2]
11:31:43: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
11:31:43: [<ffffffff81083e1c>] ? lock_timer_base+0x3c/0x70
11:31:43: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:31:43: [<ffffffffa03e5a58>] kjournald2+0xb8/0x220 [jbd2]
11:31:43: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:31:43: [<ffffffffa03e59a0>] ? kjournald2+0x0/0x220 [jbd2]
11:31:43: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:31:43: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:31:43: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:31:43: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:31:43:INFO: task lfsck:28656 blocked for more than 120 seconds.
11:31:43:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:31:43:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:31:43:lfsck         D 0000000000000000     0 28656      2 0x00000080
11:31:43: ffff88006bd19950 0000000000000046 ffff88006bd198f0 ffffffff810546b9
11:31:43: ffff88006bd19920 0000000300000001 0000000000001000 ffff88007946d0c8
11:31:43: ffff8800296825f8 ffff88006bd19fd8 000000000000fbc8 ffff8800296825f8
11:31:43:Call Trace:
11:31:43: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
11:31:43: [<ffffffffa03de08a>] start_this_handle+0x25a/0x480 [jbd2]
11:31:43: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:31:43: [<ffffffffa03de495>] jbd2_journal_start+0xb5/0x100 [jbd2]
11:31:43: [<ffffffffa0436706>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
11:31:43: [<ffffffffa0d1e91f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
11:31:43: [<ffffffffa0e510c5>] ? dt_declare_insert+0x95/0x1a0 [lfsck]
11:31:43: [<ffffffffa0e51e18>] lfsck_namespace_trace_update+0x458/0xa70 [lfsck]
11:31:43: [<ffffffffa0e59768>] lfsck_namespace_exec_oit+0x218/0xd70 [lfsck]
11:31:43: [<ffffffffa0e47100>] lfsck_exec_oit+0x70/0xcf0 [lfsck]
11:31:43: [<ffffffffa0a4efea>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
11:31:43: [<ffffffffa0e49cea>] lfsck_master_oit_engine+0x165a/0x1c00 [lfsck]
11:31:43: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
11:31:43: [<ffffffffa0e4adc0>] lfsck_master_engine+0xb30/0x13e0 [lfsck]
11:31:43: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
11:31:43: [<ffffffffa0e4a290>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
11:31:43: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:31:43: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:31:43: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:31:43: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:31:43:INFO: task lfsck_namespace:28658 blocked for more than 120 seconds.
11:31:43:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:31:43:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:33:45:lfsck_namespa D 0000000000000001     0 28658      2 0x00000080
11:33:45: ffff88004d62bb50 0000000000000046 ffff88004d62bb10 ffffffffa0d299e3
11:33:45: ffff88004d62bae0 ffffffffa0436678 ffff88004bbcbc40 ffff88005f1bb940
11:33:45: ffff88007d0df058 ffff88004d62bfd8 000000000000fbc8 ffff88007d0df058
11:33:45:Call Trace:
11:33:45: [<ffffffffa0d299e3>] ? osd_xattr_set+0x393/0x470 [osd_ldiskfs]
11:33:45: [<ffffffffa0436678>] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs]
11:33:45: [<ffffffffa0fbc92d>] ? lod_xattr_set_internal+0x1bd/0x420 [lod]
11:33:45: [<ffffffff8152bd85>] rwsem_down_failed_common+0x95/0x1d0
11:33:45: [<ffffffff8152bee3>] rwsem_down_write_failed+0x23/0x30
11:33:45: [<ffffffff8128fc03>] call_rwsem_down_write_failed+0x13/0x20
11:33:45: [<ffffffff8152b3e2>] ? down_write+0x32/0x40
11:33:45: [<ffffffffa0e558fa>] lfsck_namespace_assistant_handler_p1+0x103a/0x15d0 [lfsck]
11:33:45: [<ffffffffa0e4baff>] lfsck_assistant_engine+0x48f/0x1c50 [lfsck]
11:33:45: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
11:33:45: [<ffffffffa0e4b670>] ? lfsck_assistant_engine+0x0/0x1c50 [lfsck]
11:33:45: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:33:45: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:33:45: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:33:45: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:33:45:INFO: task lctl:28706 blocked for more than 120 seconds.
11:33:45:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:33:45:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:33:45:lctl          D 0000000000000000     0 28706  28705 0x00000080
11:33:45: ffff88006d547b90 0000000000000086 0000000000000000 ffff880000034c28
11:33:45: 0000000000000000 ffff88007efa2000 ffff88005fb87598 ffff88006d547b28
11:33:45: ffff88006ec93058 ffff88006d547fd8 000000000000fbc8 ffff88006ec93058
11:33:45:Call Trace:
11:33:45: [<ffffffff8152bd85>] rwsem_down_failed_common+0x95/0x1d0
11:33:45: [<ffffffff8152bf16>] rwsem_down_read_failed+0x26/0x30
11:33:45: [<ffffffff8128fbd4>] call_rwsem_down_read_failed+0x14/0x30
11:33:45: [<ffffffff8152b414>] ? down_read+0x24/0x30
11:33:45: [<ffffffffa0e4ee55>] lfsck_namespace_dump+0x45/0x640 [lfsck]
11:33:45: [<ffffffffa107fa3b>] ? osp_key_init+0x6b/0x190 [osp]
11:33:45: [<ffffffffa05e6c0f>] ? keys_fill+0x6f/0x190 [obdclass]
11:33:45: [<ffffffffa05c2540>] ? lprocfs_single_release+0x0/0x10 [obdclass]
11:33:45: [<ffffffffa05eb2e3>] ? lu_context_init+0xa3/0x240 [obdclass]
11:33:45: [<ffffffffa0e457b3>] lfsck_dump+0x153/0x480 [lfsck]
11:33:45: [<ffffffffa102a2a3>] mdd_lfsck_namespace_seq_show+0x23/0x60 [mdd]
11:33:45: [<ffffffff811aebd2>] seq_read+0xf2/0x400
11:33:45: [<ffffffff811f431e>] proc_reg_read+0x7e/0xc0
11:33:45: [<ffffffff81189a95>] vfs_read+0xb5/0x1a0
11:33:45: [<ffffffff81189bd1>] sys_read+0x51/0x90
11:33:45: [<ffffffff810e204e>] ? __audit_syscall_exit+0x25e/0x290
11:33:45: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
11:33:45:INFO: task jbd2/dm-0-8:26472 blocked for more than 120 seconds.
11:33:45:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:33:45:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:33:45:jbd2/dm-0-8   D 0000000000000001     0 26472      2 0x00000080
11:33:45: ffff88007a927d20 0000000000000046 ffff88006ecb36d0 ffff88006ecb36c0
11:33:45: ffff88007a3f8a40 ffff8800023168e8 000000000003c174 ffff880079472080
11:33:45: ffff880079472638 ffff88007a927fd8 000000000000fbc8 ffff880079472638
11:33:45:Call Trace:
11:33:45: [<ffffffff8109b2ce>] ? prepare_to_wait+0x4e/0x80
11:33:45: [<ffffffffa03df80f>] jbd2_journal_commit_transaction+0x19f/0x1500 [jbd2]
11:33:45: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
11:33:45: [<ffffffff81083e1c>] ? lock_timer_base+0x3c/0x70
11:33:45: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:33:45: [<ffffffffa03e5a58>] kjournald2+0xb8/0x220 [jbd2]
11:33:45: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:33:45: [<ffffffffa03e59a0>] ? kjournald2+0x0/0x220 [jbd2]
11:33:45: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:33:45: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:33:45: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:33:45: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:33:45:INFO: task lfsck:28656 blocked for more than 120 seconds.
11:33:45:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:33:45:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:33:45:lfsck         D 0000000000000000     0 28656      2 0x00000080
11:33:45: ffff88006bd19950 0000000000000046 ffff88006bd198f0 ffffffff810546b9
11:33:45: ffff88006bd19920 0000000300000001 0000000000001000 ffff88007946d0c8
11:35:46: ffff8800296825f8 ffff88006bd19fd8 000000000000fbc8 ffff8800296825f8
11:35:46:Call Trace:
11:35:46: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
11:35:46: [<ffffffffa03de08a>] start_this_handle+0x25a/0x480 [jbd2]
11:35:46: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
11:35:46: [<ffffffffa03de495>] jbd2_journal_start+0xb5/0x100 [jbd2]
11:35:46: [<ffffffffa0436706>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
11:35:46: [<ffffffffa0d1e91f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
11:35:46: [<ffffffffa0e510c5>] ? dt_declare_insert+0x95/0x1a0 [lfsck]
11:35:46: [<ffffffffa0e51e18>] lfsck_namespace_trace_update+0x458/0xa70 [lfsck]
11:35:46: [<ffffffffa0e59768>] lfsck_namespace_exec_oit+0x218/0xd70 [lfsck]
11:35:46: [<ffffffffa0e47100>] lfsck_exec_oit+0x70/0xcf0 [lfsck]
11:35:46: [<ffffffffa0a4efea>] ? fld_cache_lookup+0x3a/0x1e0 [fld]
11:35:46: [<ffffffffa0e49cea>] lfsck_master_oit_engine+0x165a/0x1c00 [lfsck]
11:35:46: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
11:35:46: [<ffffffffa0e4adc0>] lfsck_master_engine+0xb30/0x13e0 [lfsck]
11:35:46: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
11:35:46: [<ffffffffa0e4a290>] ? lfsck_master_engine+0x0/0x13e0 [lfsck]
11:35:46: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:35:46: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:35:46: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:35:46: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:35:46:INFO: task lfsck_namespace:28658 blocked for more than 120 seconds.
11:35:46:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:35:46:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:35:46:lfsck_namespa D 0000000000000001     0 28658      2 0x00000080
11:35:46: ffff88004d62bb50 0000000000000046 ffff88004d62bb10 ffffffffa0d299e3
11:35:46: ffff88004d62bae0 ffffffffa0436678 ffff88004bbcbc40 ffff88005f1bb940
11:35:46: ffff88007d0df058 ffff88004d62bfd8 000000000000fbc8 ffff88007d0df058
11:35:46:Call Trace:
11:35:46: [<ffffffffa0d299e3>] ? osd_xattr_set+0x393/0x470 [osd_ldiskfs]
11:35:46: [<ffffffffa0436678>] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs]
11:35:46: [<ffffffffa0fbc92d>] ? lod_xattr_set_internal+0x1bd/0x420 [lod]
11:35:46: [<ffffffff8152bd85>] rwsem_down_failed_common+0x95/0x1d0
11:35:46: [<ffffffff8152bee3>] rwsem_down_write_failed+0x23/0x30
11:35:46: [<ffffffff8128fc03>] call_rwsem_down_write_failed+0x13/0x20
11:35:46: [<ffffffff8152b3e2>] ? down_write+0x32/0x40
11:35:46: [<ffffffffa0e558fa>] lfsck_namespace_assistant_handler_p1+0x103a/0x15d0 [lfsck]
11:35:46: [<ffffffffa0e4baff>] lfsck_assistant_engine+0x48f/0x1c50 [lfsck]
11:35:46: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
11:35:46: [<ffffffffa0e4b670>] ? lfsck_assistant_engine+0x0/0x1c50 [lfsck]
11:35:46: [<ffffffff8109abf6>] kthread+0x96/0xa0
11:35:46: [<ffffffff8100c20a>] child_rip+0xa/0x20
11:35:46: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
11:35:46: [<ffffffff8100c200>] ? child_rip+0x0/0x20
11:35:46:INFO: task lctl:28706 blocked for more than 120 seconds.
11:35:46:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1
11:35:46:"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
11:35:46:lctl          D 0000000000000000     0 28706  28705 0x00000080
11:35:46: ffff88006d547b90 0000000000000086 0000000000000000 ffff880000034c28
11:35:46: 0000000000000000 ffff88007efa2000 ffff88005fb87598 ffff88006d547b28
11:35:46: ffff88006ec93058 ffff88006d547fd8 000000000000fbc8 ffff88006ec93058
11:35:46:Call Trace:
11:35:46: [<ffffffff8152bd85>] rwsem_down_failed_common+0x95/0x1d0
11:35:46: [<ffffffff8152bf16>] rwsem_down_read_failed+0x26/0x30
11:35:46: [<ffffffff8128fbd4>] call_rwsem_down_read_failed+0x14/0x30
11:35:46: [<ffffffff8152b414>] ? down_read+0x24/0x30
11:35:46: [<ffffffffa0e4ee55>] lfsck_namespace_dump+0x45/0x640 [lfsck]
11:35:46: [<ffffffffa107fa3b>] ? osp_key_init+0x6b/0x190 [osp]
11:35:46: [<ffffffffa05e6c0f>] ? keys_fill+0x6f/0x190 [obdclass]
11:35:46: [<ffffffffa05c2540>] ? lprocfs_single_release+0x0/0x10 [obdclass]
11:35:46: [<ffffffffa05eb2e3>] ? lu_context_init+0xa3/0x240 [obdclass]
11:35:46: [<ffffffffa0e457b3>] lfsck_dump+0x153/0x480 [lfsck]
11:35:46: [<ffffffffa102a2a3>] mdd_lfsck_namespace_seq_show+0x23/0x60 [mdd]
11:35:46: [<ffffffff811aebd2>] seq_read+0xf2/0x400
11:35:46: [<ffffffff811f431e>] proc_reg_read+0x7e/0xc0
11:35:46: [<ffffffff81189a95>] vfs_read+0xb5/0x1a0
11:35:46: [<ffffffff81189bd1>] sys_read+0x51/0x90
11:35:46: [<ffffffff810e204e>] ? __audit_syscall_exit+0x25e/0x290
11:35:46: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
11:35:46:INFO: task jbd2/dm-0-8:26472 blocked for more than 120 seconds.
11:35:46:      Not tainted 2.6.32-431.29.2.el6_lustre.g5d1aa14.x86_64 #1

Info required for matching: sanity-lfsck 9b

Whamcloud Community JIRA

[LU-5672] sanity-lfsck test_9b: journal deadlock with lfsck thread