https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4105 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/758c2064-3456-11e3-9356-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/758c2064-3456-11e3-9356-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>client console shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:50:18:Lustre: DEBUG MARKER: == racer test 1: racer on clients: client-13vm1.lab.whamcloud.com,client-13vm2 DURATION=900 == 08:49:53 (1381592993) 08:50:18:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer 08:50:19:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer 08:54:39:Lustre: lustre-OST0004-osc-ffff880032581c00: Connection to lustre-OST0004 (at 10.10.4.212@tcp) was lost; in progress operations using this service will wait for recovery to complete 08:54:42:LustreError: 11-0: lustre-OST0004-osc-ffff880032581c00: Communicating with 10.10.4.212@tcp, operation ldlm_enqueue failed with -107. 08:54:44:LustreError: 167-0: lustre-OST0004-osc-ffff880032581c00: This client was evicted by lustre-OST0004; in progress operations using this service will fail. 08:54:44:Lustre: 26160:0:(llite_lib.c:2484:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.10.4.211@tcp:/lustre/fid: [0x200000403:0x158b:0x0]/ may get corrupted (rc -108) 08:54:45:Lustre: 26161:0:(llite_lib.c:2484:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.10.4.211@tcp:/lustre/fid: [0x200000402:0x185b:0x0]//racer/11/11/11 may get corrupted (rc -108) 08:54:45:Lustre: lustre-OST0004-osc-ffff880032581c00: Connection restored to lustre-OST0004 (at 10.10.4.212@tcp) 09:01:24:INFO: task dir_create.sh:7007 blocked for more than 120 seconds. 09:01:25:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:01:26:dir_create.sh D 0000000000000000 0 7007 6999 0x00000080 09:01:27: ffff880029a7fbd8 0000000000000086 525971bf00000000 0000000000057b80 09:01:27: 00001b5f00000000 00000d1d00000000 ffff880029a7fc38 ffffffffa08e9d3d 09:01:27: ffff88004badf098 ffff880029a7ffd8 000000000000fb88 ffff88004badf098 09:01:27:Call Trace: 09:01:27: [&lt;ffffffffa08e9d3d&gt;] ? ll_revalidate_it+0x48d/0x1b20 [lustre] 09:01:27: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:01:27: [&lt;ffffffff8119af58&gt;] ? __d_lookup+0xd8/0x150 09:01:27: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:01:28: [&lt;ffffffff811907ab&gt;] do_lookup+0x11b/0x230 09:01:28: [&lt;ffffffff81190acd&gt;] __link_path_walk+0x20d/0x1030 09:01:28: [&lt;ffffffff81191b7a&gt;] path_walk+0x6a/0xe0 09:01:28: [&lt;ffffffff81192d91&gt;] do_filp_open+0x201/0xdc0 09:01:29: [&lt;ffffffff8104759c&gt;] ? __do_page_fault+0x1ec/0x480 09:01:30: [&lt;ffffffff81014f79&gt;] ? init_fpu+0x59/0xc0 09:01:30: [&lt;ffffffff8119f922&gt;] ? alloc_fd+0x92/0x160 09:01:30: [&lt;ffffffff8117e249&gt;] do_sys_open+0x69/0x140 09:01:31: [&lt;ffffffff8117e360&gt;] sys_open+0x20/0x30 09:01:31: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:01:31:INFO: task dir_create.sh:7016 blocked for more than 120 seconds. 09:01:31:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:01:32:dir_create.sh D 0000000000000000 0 7016 6999 0x00000080 09:01:32: ffff88007cdefbd8 0000000000000086 525971bf00000000 000000000009639c 09:01:32: 00001b6800000000 00000d1d00000000 ffff88007cdefc38 ffffffffa08e9d3d 09:01:32: ffff88007cdedab8 ffff88007cdeffd8 000000000000fb88 ffff88007cdedab8 09:01:32:Call Trace: 09:01:32: [&lt;ffffffffa08e9d3d&gt;] ? ll_revalidate_it+0x48d/0x1b20 [lustre] 09:01:33: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:01:34: [&lt;ffffffff8119af58&gt;] ? __d_lookup+0xd8/0x150 09:01:34: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:01:35: [&lt;ffffffff811907ab&gt;] do_lookup+0x11b/0x230 09:01:35: [&lt;ffffffff81190acd&gt;] __link_path_walk+0x20d/0x1030 09:01:35: [&lt;ffffffff81191b7a&gt;] path_walk+0x6a/0xe0 09:01:36: [&lt;ffffffff81192d91&gt;] do_filp_open+0x201/0xdc0 09:01:36: [&lt;ffffffff8104759c&gt;] ? __do_page_fault+0x1ec/0x480 09:01:36: [&lt;ffffffff81014f79&gt;] ? init_fpu+0x59/0xc0 09:01:38: [&lt;ffffffff8119f922&gt;] ? alloc_fd+0x92/0x160 09:01:38: [&lt;ffffffff8117e249&gt;] do_sys_open+0x69/0x140 09:01:39: [&lt;ffffffff8117e360&gt;] sys_open+0x20/0x30 09:01:39: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:01:39:INFO: task rm:14589 blocked for more than 120 seconds. 09:01:39:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:01:39:rm D 0000000000000000 0 14589 7055 0x00000080 09:01:40: ffff880068037e08 0000000000000086 ffff8800103e4040 ffff8800103e4040 09:01:40: ffff8800103e4040 ffff880068169040 ffff8800103e4040 0000001000000286 09:01:40: ffff8800103e45f8 ffff880068037fd8 000000000000fb88 ffff8800103e45f8 09:01:40:Call Trace: 09:01:40: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:01:41: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:01:41: [&lt;ffffffff81192367&gt;] do_rmdir+0xb7/0x120 09:01:42: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:01:43: [&lt;ffffffff811923fd&gt;] sys_unlinkat+0x2d/0x40 09:01:43: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:01:43:INFO: task mv:14594 blocked for more than 120 seconds. 09:01:44:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:01:46:mv D 0000000000000000 0 14594 7009 0x00000080 09:01:46: ffff8800376c7d08 0000000000000082 ffff8800376c7d18 ffffffff81190a15 09:01:47: 0000003f1ec64000 ffffea00016d5268 ffff8800376c7d08 ffff880018523540 09:01:47: ffff880018523af8 ffff8800376c7fd8 000000000000fb88 ffff880018523af8 09:01:47:Call Trace: 09:01:47: [&lt;ffffffff81190a15&gt;] ? __link_path_walk+0x155/0x1030 09:01:50: [&lt;ffffffff811a20d0&gt;] ? mntput_no_expire+0x30/0x110 09:01:50: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:01:50: [&lt;ffffffff8118f541&gt;] ? path_put+0x31/0x40 09:01:51: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:01:51: [&lt;ffffffff8118e9b3&gt;] lock_rename+0x63/0xe0 09:01:51: [&lt;ffffffff81191f9f&gt;] sys_renameat+0x11f/0x240 09:01:52: [&lt;ffffffff81186b64&gt;] ? cp_new_stat+0xe4/0x100 09:01:53: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:01:53: [&lt;ffffffff811920db&gt;] sys_rename+0x1b/0x20 09:01:53: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:01:53:INFO: task mv:14603 blocked for more than 120 seconds. 09:01:54:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:01:54:mv D 0000000000000000 0 14603 7038 0x00000080 09:01:54: ffff88005fc1dd08 0000000000000082 ffff88005fc1dd18 ffffffff81190a15 09:01:54: 0000003f1ec64000 ffffea00016d5268 ffff88005fc1dd08 ffff88002d8d4040 09:01:54: ffff88002d8d45f8 ffff88005fc1dfd8 000000000000fb88 ffff88002d8d45f8 09:01:54:Call Trace: 09:01:55: [&lt;ffffffff81190a15&gt;] ? __link_path_walk+0x155/0x1030 09:01:56: [&lt;ffffffff811a20d0&gt;] ? mntput_no_expire+0x30/0x110 09:01:57: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:01:57: [&lt;ffffffff8118f541&gt;] ? path_put+0x31/0x40 09:01:58: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:01:58: [&lt;ffffffff8118e98f&gt;] lock_rename+0x3f/0xe0 09:01:58: [&lt;ffffffff81191f9f&gt;] sys_renameat+0x11f/0x240 09:01:58: [&lt;ffffffff81186b64&gt;] ? cp_new_stat+0xe4/0x100 09:01:58: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:01:58: [&lt;ffffffff811920db&gt;] sys_rename+0x1b/0x20 09:01:59: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:01:59:INFO: task ln:14613 blocked for more than 120 seconds. 09:02:00:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:02:01:ln D 0000000000000000 0 14613 7011 0x00000080 09:02:02: ffff88005c457dd8 0000000000000082 ffff88005e721bb8 0000000000000000 09:02:02: ffff88005c457d88 ffff88007c2f4ae0 ffff88007c2f4ae0 ffff88007c2f4ae0 09:02:03: ffff88007c2f5098 ffff88005c457fd8 000000000000fb88 ffff88007c2f5098 09:02:03:Call Trace: 09:02:03: [&lt;ffffffff811a20d0&gt;] ? mntput_no_expire+0x30/0x110 09:02:03: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:02:03: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:02:03: [&lt;ffffffff8118ebb0&gt;] lookup_create+0x30/0xd0 09:02:04: [&lt;ffffffff811928d9&gt;] sys_symlinkat+0x99/0x120 09:02:05: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:02:07: [&lt;ffffffff81192976&gt;] sys_symlink+0x16/0x20 09:02:07: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:02:08:INFO: task rm:14615 blocked for more than 120 seconds. 09:02:08:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:02:09:rm D 0000000000000000 0 14615 7046 0x00000080 09:02:09: ffff88006088de08 0000000000000086 ffff880019cb7540 ffff880019cb7540 09:02:10: ffff880019cb7540 ffff88005dd3c1b8 ffff880019cb7540 000000108119cfaf 09:02:11: ffff880019cb7af8 ffff88006088dfd8 000000000000fb88 ffff880019cb7af8 09:02:12:Call Trace: 09:02:13: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:02:13: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:02:14: [&lt;ffffffff81192367&gt;] do_rmdir+0xb7/0x120 09:02:14: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:02:15: [&lt;ffffffff811923fd&gt;] sys_unlinkat+0x2d/0x40 09:02:15: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:02:15:INFO: task rm:14625 blocked for more than 120 seconds. 09:02:16:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:02:16:rm D 0000000000000000 0 14625 7026 0x00000080 09:02:17: ffff8800376c5e08 0000000000000086 ffff88005e07f500 ffff88005e07f500 09:02:17: ffff88005e07f500 ffff8800102690c0 ffff88005e07f500 0000001000000286 09:02:17: ffff88005e07fab8 ffff8800376c5fd8 000000000000fb88 ffff88005e07fab8 09:02:17:Call Trace: 09:02:19: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:02:20: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:02:20: [&lt;ffffffff81192367&gt;] do_rmdir+0xb7/0x120 09:02:20: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:02:20: [&lt;ffffffff811923fd&gt;] sys_unlinkat+0x2d/0x40 09:02:20: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:02:21:INFO: task mv:14630 blocked for more than 120 seconds. 09:02:21:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:02:22:mv D 0000000000000000 0 14630 7018 0x00000080 09:02:24: ffff8800670b5d08 0000000000000082 ffff8800670b5d18 ffffffff81190a15 09:02:25: 0000003f1ec64000 ffffea00016d5268 ffff8800670b5c98 ffff8800103dd540 09:02:25: ffff8800103ddaf8 ffff8800670b5fd8 000000000000fb88 ffff8800103ddaf8 09:02:26:Call Trace: 09:02:26: [&lt;ffffffff81190a15&gt;] ? __link_path_walk+0x155/0x1030 09:02:26: [&lt;ffffffff811a20d0&gt;] ? mntput_no_expire+0x30/0x110 09:02:28: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:02:29: [&lt;ffffffff8118f541&gt;] ? path_put+0x31/0x40 09:02:30: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:02:30: [&lt;ffffffff8118e98f&gt;] lock_rename+0x3f/0xe0 09:02:30: [&lt;ffffffff81191f9f&gt;] sys_renameat+0x11f/0x240 09:02:30: [&lt;ffffffff81186b64&gt;] ? cp_new_stat+0xe4/0x100 09:02:30: [&lt;ffffffff81186e3e&gt;] ? vfs_lstat+0x1e/0x20 09:02:30: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:02:31: [&lt;ffffffff811920db&gt;] sys_rename+0x1b/0x20 09:02:31: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:02:32:INFO: task ls:14643 blocked for more than 120 seconds. 09:02:33:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:02:34:ls D 0000000000000000 0 14643 7031 0x00000080 09:02:34: ffff88005d631748 0000000000000086 ffff88005d631728 ffffffffa0807116 09:02:34: 0000000000000000 ffffffffa086b240 ffff88005f9e6400 ffffffffa07e04c0 09:02:35: ffff88007a273058 ffff88005d631fd8 000000000000fb88 ffff88007a273058 09:02:35:Call Trace: 09:02:35: [&lt;ffffffffa0807116&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] 09:02:35: [&lt;ffffffffa07e04c0&gt;] ? lustre_msg_string+0x0/0x290 [ptlrpc] 09:02:36: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:02:36: [&lt;ffffffffa0493167&gt;] ? mdc_open_pack+0x247/0x260 [mdc] 09:02:37: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 09:02:38: [&lt;ffffffffa0497a7d&gt;] mdc_enqueue+0x22d/0x1a10 [mdc] 09:02:38: [&lt;ffffffffa049945d&gt;] mdc_intent_lock+0x1fd/0x64a [mdc] 09:02:39: [&lt;ffffffffa0932740&gt;] ? ll_md_blocking_ast+0x0/0x790 [lustre] 09:02:39: [&lt;ffffffffa07bf520&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] 09:02:39: [&lt;ffffffffa0495d71&gt;] ? mdc_lock_match+0xb1/0x160 [mdc] 09:02:39: [&lt;ffffffffa03d85a3&gt;] lmv_intent_open+0x213/0x8d0 [lmv] 09:02:41: [&lt;ffffffffa0932740&gt;] ? ll_md_blocking_ast+0x0/0x790 [lustre] 09:02:42: [&lt;ffffffffa03be7cb&gt;] ? lmv_lock_match+0x1bb/0x470 [lmv] 09:02:42: [&lt;ffffffffa062ba15&gt;] ? class_handle2object+0x95/0x190 [obdclass] 09:02:42: [&lt;ffffffffa03d8f1b&gt;] lmv_intent_lock+0x2bb/0x380 [lmv] 09:02:42: [&lt;ffffffffa0932740&gt;] ? ll_md_blocking_ast+0x0/0x790 [lustre] 09:02:42: [&lt;ffffffffa0931e8d&gt;] ? ll_i2gids+0x3d/0xd0 [lustre] 09:02:42: [&lt;ffffffffa0918c9d&gt;] ? ll_prep_md_op_data+0x10d/0x3b0 [lustre] 09:02:43: [&lt;ffffffffa08e9b25&gt;] ll_revalidate_it+0x275/0x1b20 [lustre] 09:02:43: [&lt;ffffffffa0932740&gt;] ? ll_md_blocking_ast+0x0/0x790 [lustre] 09:02:43: [&lt;ffffffffa08eb503&gt;] ll_revalidate_nd+0x133/0x3e0 [lustre] 09:02:43: [&lt;ffffffff811906f6&gt;] do_lookup+0x66/0x230 09:02:44: [&lt;ffffffff81190ff4&gt;] __link_path_walk+0x734/0x1030 09:02:44: [&lt;ffffffffa03c511d&gt;] ? lmv_clear_open_replay_data+0x2ad/0x6d0 [lmv] 09:02:45: [&lt;ffffffffa07d06b0&gt;] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] 09:02:45: [&lt;ffffffff81191b7a&gt;] path_walk+0x6a/0xe0 09:02:45: [&lt;ffffffff81191d4b&gt;] do_path_lookup+0x5b/0xa0 09:02:45: [&lt;ffffffff81182830&gt;] ? get_empty_filp+0xa0/0x180 09:02:46: [&lt;ffffffff81192c8b&gt;] do_filp_open+0xfb/0xdc0 09:02:46: [&lt;ffffffff811a20d0&gt;] ? mntput_no_expire+0x30/0x110 09:02:46: [&lt;ffffffff8119f922&gt;] ? alloc_fd+0x92/0x160 09:02:46: [&lt;ffffffff8117e249&gt;] do_sys_open+0x69/0x140 09:02:46: [&lt;ffffffff8100c535&gt;] ? math_state_restore+0x45/0x60 09:02:46: [&lt;ffffffff8117e360&gt;] sys_open+0x20/0x30 09:02:47: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 09:02:47:Lustre: lustre-MDT0000-mdc-ffff880032581c00: Connection to lustre-MDT0000 (at 10.10.4.211@tcp) was lost; in progress operations using this service will wait for recovery to complete 09:02:47:LustreError: 14642:0:(ldlm_request.c:134:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1381593535, 200s ago), entering recovery for lustre-MDT0000_UUID@10.10.4.211@tcp ns: lustre-MDT0000-mdc-ffff880032581c00 lock: ffff880067089b40/0x19bd7e0c7fd16370 lrc: 4/1,0 mode: --/PR res: [0x200000400:0x1:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x31861536cea3e045 expref: -99 pid: 14642 timeout: 0 lvb_type: 0 </pre> </div></div> client and server: lustre-b2_5 RHEL6 build #2 LU-4105

Failure on test suite racer test_1

Bug Blocker Resolved Duplicate WC Triage Maloo Mon, 14 Oct 2013 18:51:09 +0000 Tue, 21 Oct 2014 21:24:56 +0000 Tue, 21 Oct 2014 21:24:56 +0000 Lustre 2.5.0 Lustre 2.7.0 Lustre 2.5.3 0 6 <p>A similar error seen in SLES11 SP3 client, I am not sure if this is the same one:</p> <p><a href="https://maloo.whamcloud.com/test_sets/9aefe8ee-381c-11e3-844f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9aefe8ee-381c-11e3-844f-52540035b04c</a></p> <p>another ticket to track this issue <a href="https://jira.whamcloud.com/browse/LU-4133" title="Failure on test suite racer test_1" class="issue-link" data-issue-key="LU-4133"><del>LU-4133</del></a></p> <p>hit same error here:<br/> <a href="https://maloo.whamcloud.com/test_sets/931f9f5c-80dc-11e3-81ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/931f9f5c-80dc-11e3-81ba-52540035b04c</a></p> <p>server and client: lustre-master build #1837 RHEL6 ldiskfs DNE mode</p> <p>client 1 console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:03:08:INFO: task mv:22208 blocked for more than 120 seconds. 00:03:09:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 00:03:10:mv D 0000000000000001 0 22208 3780 0x00000080 00:03:10: ffff8800536c7d08 0000000000000082 ffff8800536c7d18 ffffffff81190a45 00:03:10: ffff880036d42500 ffff8800483bffa0 ffffffff8100bb8e ffff8800536c7d08 00:03:11: ffff88006c4a2638 ffff8800536c7fd8 000000000000fb88 ffff88006c4a2638 00:03:11:Call Trace: 00:03:11: [&lt;ffffffff81190a45&gt;] ? __link_path_walk+0x155/0x1030 00:03:11: [&lt;ffffffff8100bb8e&gt;] ? apic_timer_interrupt+0xe/0x20 00:03:13: [&lt;ffffffff810521eb&gt;] ? mutex_spin_on_owner+0x9b/0xc0 00:03:14: [&lt;ffffffff8150f81e&gt;] __mutex_lock_slowpath+0x13e/0x180 00:03:14: [&lt;ffffffff8118f571&gt;] ? path_put+0x31/0x40 00:03:14: [&lt;ffffffff8150f6bb&gt;] mutex_lock+0x2b/0x50 00:03:14: [&lt;ffffffff8118ea37&gt;] lock_rename+0xb7/0xe0 00:03:14: [&lt;ffffffff81191fcf&gt;] sys_renameat+0x11f/0x240 00:03:14: [&lt;ffffffff81186b94&gt;] ? cp_new_stat+0xe4/0x100 00:03:15: [&lt;ffffffff81186ea6&gt;] ? sys_newlstat+0x36/0x50 00:03:15: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 00:03:15: [&lt;ffffffff8119210b&gt;] sys_rename+0x1b/0x20 00:03:15: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 00:03:15:INFO: task rm:23013 blocked for more than 120 seconds. 00:03:15:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 00:03:16:rm D 0000000000000001 0 23013 3800 0x00000080 00:03:16: ffff8800483a1e08 0000000000000086 ffff88006ce8c080 ffff88006ce8c080 00:03:17: ffff88006ce8c080 ffff88007ba2d000 ffff88006ce8c080 00000010a0a57caf 00:03:17: ffff88006ce8c638 ffff8800483a1fd8 000000000000fb88 ffff88006ce8c638 00:03:18:Call Trace: 00:03:18: [&lt;ffffffff8118f571&gt;] ? path_put+0x31/0x40 00:03:18: [&lt;ffffffff8150f81e&gt;] __mutex_lock_slowpath+0x13e/0x180 00:03:18: [&lt;ffffffff8150f6bb&gt;] mutex_lock+0x2b/0x50 00:03:18: [&lt;ffffffff811921b9&gt;] do_unlinkat+0xa9/0x1b0 00:03:18: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 00:03:19: [&lt;ffffffff81192422&gt;] sys_unlinkat+0x22/0x40 00:03:20: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 00:03:20:INFO: task mv:23427 blocked for more than 120 seconds. 00:03:20:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 00:03:20:mv D 0000000000000001 0 23427 3790 0x00000080 00:03:21: ffff880041c23d08 0000000000000086 ffff880041c23d18 ffffffff81190a45 00:03:21: 00007fae6043f000 ffffea0000c33ff0 ffff880041c23c98 ffff88006edd8080 00:03:21: ffff88006edd8638 ffff880041c23fd8 000000000000fb88 ffff88006edd8638 00:03:22:Call Trace: 00:03:23: [&lt;ffffffff81190a45&gt;] ? __link_path_walk+0x155/0x1030 00:03:23: [&lt;ffffffff811a2100&gt;] ? mntput_no_expire+0x30/0x110 00:03:23: [&lt;ffffffff8150f81e&gt;] __mutex_lock_slowpath+0x13e/0x180 00:03:24: [&lt;ffffffff8118f571&gt;] ? path_put+0x31/0x40 00:03:24: [&lt;ffffffff8150f6bb&gt;] mutex_lock+0x2b/0x50 00:03:25: [&lt;ffffffff8118e9bf&gt;] lock_rename+0x3f/0xe0 00:03:25: [&lt;ffffffff81191fcf&gt;] sys_renameat+0x11f/0x240 00:03:25: [&lt;ffffffff81186b94&gt;] ? cp_new_stat+0xe4/0x100 00:03:25: [&lt;ffffffff81186e6e&gt;] ? vfs_lstat+0x1e/0x20 00:03:26: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 00:03:27: [&lt;ffffffff8119210b&gt;] sys_rename+0x1b/0x20 00:03:27: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>another instance:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/ba82eb98-14ee-11e4-bb6a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ba82eb98-14ee-11e4-bb6a-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:36:20:LustreError: 12291:0:(file.c:3105:ll_migrate()) lustre: migrate 5 , but fid [0x0:0x0:0x0] is insane 04:36:21:LustreError: 12993:0:(file.c:3105:ll_migrate()) lustre: migrate 1 , but fid [0x0:0x0:0x0] is insane 04:36:21:LustreError: 12993:0:(file.c:3105:ll_migrate()) Skipped 4 previous similar messages 04:36:22:LustreError: 14629:0:(file.c:3105:ll_migrate()) lustre: migrate 17 , but fid [0x0:0x0:0x0] is insane 04:36:22:LustreError: 14629:0:(file.c:3105:ll_migrate()) Skipped 5 previous similar messages 04:36:22:LustreError: 15942:0:(file.c:3105:ll_migrate()) lustre: migrate 7 , but fid [0x0:0x0:0x0] is insane 04:36:22:LustreError: 15942:0:(file.c:3105:ll_migrate()) Skipped 2 previous similar messages 04:36:24:LustreError: 16166:0:(file.c:3105:ll_migrate()) lustre: migrate 4 , but fid [0x0:0x0:0x0] is insane 04:36:24:LustreError: 16166:0:(file.c:3105:ll_migrate()) Skipped 5 previous similar messages 04:36:24:LustreError: 16510:0:(file.c:3105:ll_migrate()) lustre: migrate 13 , but fid [0x0:0x0:0x0] is insane 04:36:24:LustreError: 16510:0:(file.c:3105:ll_migrate()) Skipped 6 previous similar messages 04:36:25:LustreError: 20445:0:(file.c:3105:ll_migrate()) lustre: migrate 7 , but fid [0x0:0x0:0x0] is insane 04:36:25:LustreError: 20445:0:(file.c:3105:ll_migrate()) Skipped 13 previous similar messages 04:36:25:LustreError: 28207:0:(file.c:3105:ll_migrate()) lustre: migrate 0 , but fid [0x0:0x0:0x0] is insane 04:36:25:LustreError: 28207:0:(file.c:3105:ll_migrate()) Skipped 27 previous similar messages 04:36:27:LustreError: 2474:0:(lmv_intent.c:239:lmv_revalidate_slaves()) lustre-clilmv-ffff88007b1c9c00: nlink 0 &lt; 2 corrupt stripe 0 [0x3c0000403:0xbf1:0x0]:[0x3c0000403:0xbf1:0x0] 04:38:32:LustreError: 2474:0:(llite_lib.c:2419:ll_prep_inode()) new_inode -fatal: rc -5 04:38:33:LNet: Host 10.2.4.243 reset our connection while we were sending data; it may have rebooted. 04:38:33:Lustre: 2431:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1406288230/real 1406288230] req@ffff88006b84dc00 x1474599723026556/t0(0) o400-&gt;lustre-MDT0000-mdc-ffff88007b1c9c00@10.2.4.243@tcp:12/10 lens 224/224 e 0 to 1 dl 1406288414 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:33:LustreError: 166-1: MGC10.2.4.243@tcp: Connection to MGS (at 10.2.4.243@tcp) was lost; in progress operations using this service will fail 04:38:33:Lustre: lustre-MDT0001-mdc-ffff88007b1c9c00: Connection to lustre-MDT0001 (at 10.2.4.243@tcp) was lost; in progress operations using this service will wait for recovery to complete 04:38:33:Lustre: 2431:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 3 previous similar messages 04:38:33:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1406288230/real 1406288230] req@ffff88006c138400 x1474599723026628/t0(0) o250-&gt;MGC10.2.4.243@tcp@10.2.4.243@tcp:26/25 lens 400/544 e 0 to 1 dl 1406288236 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:33:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 3 previous similar messages 04:38:34:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1406288235/real 1406288235] req@ffff88006b947c00 x1474599723026648/t0(0) o38-&gt;lustre-MDT0000-mdc-ffff88007b1c9c00@10.2.4.243@tcp:12/10 lens 400/544 e 0 to 1 dl 1406288246 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:34:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 2 previous similar messages 04:38:34:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1406288240/real 0] req@ffff88006e028400 x1474599723026712/t0(0) o250-&gt;MGC10.2.4.243@tcp@10.2.4.243@tcp:26/25 lens 400/544 e 0 to 1 dl 1406288251 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:35:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 1 previous similar message 04:38:35:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1406288250/real 0] req@ffff8800700cc800 x1474599723026836/t0(0) o38-&gt;lustre-MDT0000-mdc-ffff88007b1c9c00@10.2.4.243@tcp:12/10 lens 400/544 e 0 to 1 dl 1406288266 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:35:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 2 previous similar messages 04:38:35:Lustre: 26740:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1406288091/real 1406288091] req@ffff88006d299400 x1474599722652016/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff88007aaad800@10.2.4.243@tcp:12/10 lens 576/1296 e 4 to 1 dl 1406288275 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1 04:38:35:Lustre: 26740:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 1 previous similar message 04:38:35:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1406288280/real 1406288280] req@ffff880067819400 x1474599723027200/t0(0) o250-&gt;MGC10.2.4.243@tcp@10.2.4.243@tcp:26/25 lens 400/544 e 0 to 1 dl 1406288301 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 04:38:35:Lustre: 2429:0:(client.c:1926:ptlrpc_expire_one_request()) Skipped 12 previous similar messages 04:38:36:INFO: task dir_create.sh:10472 blocked for more than 120 seconds. 04:38:36: Not tainted 2.6.32-431.20.3.el6.x86_64 #1 04:38:36:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 04:38:36:dir_create.sh D 0000000000000001 0 10472 10446 0x00000080 04:38:37: ffff880071c5fb98 0000000000000086 0000004b00000000 ffffffffa0bad040 04:38:37: 0000000000000098 0020000000000080 53d240db00000001 000000000000f756 04:38:37: ffff880072a46638 ffff880071c5ffd8 000000000000fbc8 ffff880072a46638 04:38:39:Call Trace: 04:38:39: [&lt;ffffffff81529f9e&gt;] __mutex_lock_slowpath+0x13e/0x180 04:38:40: [&lt;ffffffff811a4028&gt;] ? __d_lookup+0xd8/0x150 04:38:40: [&lt;ffffffff81529e3b&gt;] mutex_lock+0x2b/0x50 04:38:40: [&lt;ffffffff8119888b&gt;] do_lookup+0x11b/0x230 04:38:41: [&lt;ffffffff81198fe0&gt;] __link_path_walk+0x200/0x1000 04:38:41: [&lt;ffffffff8119a09a&gt;] path_walk+0x6a/0xe0 04:38:41: [&lt;ffffffff8119b87a&gt;] do_filp_open+0x1fa/0xd20 04:38:42: [&lt;ffffffff8109b39c&gt;] ? remove_wait_queue+0x3c/0x50 04:38:42: [&lt;ffffffff81016c71&gt;] ? fpu_finit+0x21/0x40 04:38:42: [&lt;ffffffff8128f70a&gt;] ? strncpy_from_user+0x4a/0x90 04:38:43: [&lt;ffffffff811a8a62&gt;] ? alloc_fd+0x92/0x160 04:38:43: [&lt;ffffffff81185ba9&gt;] do_sys_open+0x69/0x140 04:38:43: [&lt;ffffffff81185cc0&gt;] sys_open+0x20/0x30 04:38:43: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 04:38:44:INFO: task dir_create.sh:10692 blocked for more than 120 seconds. 04:38:44: Not tainted 2.6.32-431.20.3.el6.x86_64 #1 04:38:44:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 04:38:46:dir_create.sh D 0000000000000001 0 10692 10456 0x00000080 04:38:47: ffff880070315b98 0000000000000082 0000004b42547b0a ffffffffa0bad040 04:38:47: 0000000000000098 0020000000000080 53d2413500000001 00000000000c4627 04:38:47: ffff880070313ab8 ffff880070315fd8 000000000000fbc8 ffff880070313ab8 04:38:49:Call Trace: 04:38:49: [&lt;ffffffff81529f9e&gt;] __mutex_lock_slowpath+0x13e/0x180 04:38:49: [&lt;ffffffff811a4028&gt;] ? __d_lookup+0xd8/0x150 04:38:50: [&lt;ffffffff81529e3b&gt;] mutex_lock+0x2b/0x50 04:38:50: [&lt;ffffffff8119888b&gt;] do_lookup+0x11b/0x230 04:38:50: [&lt;ffffffff81198fe0&gt;] __link_path_walk+0x200/0x1000 04:38:50: [&lt;ffffffff8119a09a&gt;] path_walk+0x6a/0xe0 04:38:51: [&lt;ffffffff8119b87a&gt;] do_filp_open+0x1fa/0xd20 04:38:51: [&lt;ffffffff8109b39c&gt;] ? remove_wait_queue+0x3c/0x50 04:38:52: [&lt;ffffffff81016c71&gt;] ? fpu_finit+0x21/0x40 04:38:52: [&lt;ffffffff8128f70a&gt;] ? strncpy_from_user+0x4a/0x90 04:38:53: [&lt;ffffffff811a8a62&gt;] ? alloc_fd+0x92/0x160 04:38:53: [&lt;ffffffff81185ba9&gt;] do_sys_open+0x69/0x140 04:38:53: [&lt;ffffffff81185cc0&gt;] sys_open+0x20/0x30 04:38:53: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/80/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/80/</a><br/> Distro/Arch: RHEL6.5/x86_64</p> <p>The same failure occurred while running racer test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:34:43:Lustre: DEBUG MARKER: == racer test 1: racer on clients: shadow-23vm1,shadow-23vm5.shadow.whamcloud.com DURATION=900 == 06:19:27 (1408256367) 00:34:43:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer 00:34:43:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer 00:34:43:INFO: task dir_create.sh:16222 blocked for more than 120 seconds. 00:34:43: Not tainted 2.6.32-431.23.3.el6.x86_64 #1 00:34:43:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 00:34:43:dir_create.sh D 0000000000000001 0 16222 16215 0x00000084 00:34:43: ffff880051103b98 0000000000000086 53f04c1a00000001 0000000000061e97 00:34:43: 00003f5e00000000 00000d2700000000 ffff880051103bf8 ffffffffa0fc5d3d 00:34:43: ffff88005078c638 ffff880051103fd8 000000000000fbc8 ffff88005078c638 00:34:43:Call Trace: 00:34:43: [&lt;ffffffffa0fc5d3d&gt;] ? ll_revalidate_it+0x48d/0x1b20 [lustre] 00:34:43: [&lt;ffffffff8152a36e&gt;] __mutex_lock_slowpath+0x13e/0x180 00:34:43: [&lt;ffffffff811a4198&gt;] ? __d_lookup+0xd8/0x150 00:34:43: [&lt;ffffffff8152a20b&gt;] mutex_lock+0x2b/0x50 00:34:43: [&lt;ffffffff811989fb&gt;] do_lookup+0x11b/0x230 00:34:43: [&lt;ffffffff81199150&gt;] __link_path_walk+0x200/0x1000 00:34:43: [&lt;ffffffff8119a20a&gt;] path_walk+0x6a/0xe0 00:34:43: [&lt;ffffffff8119b9ea&gt;] do_filp_open+0x1fa/0xd20 00:34:43: [&lt;ffffffff8109b39c&gt;] ? remove_wait_queue+0x3c/0x50 00:34:43: [&lt;ffffffff81016c71&gt;] ? fpu_finit+0x21/0x40 00:34:43: [&lt;ffffffff8128f87a&gt;] ? strncpy_from_user+0x4a/0x90 00:34:43: [&lt;ffffffff811a8bd2&gt;] ? alloc_fd+0x92/0x160 00:34:43: [&lt;ffffffff81185c39&gt;] do_sys_open+0x69/0x140 00:34:43: [&lt;ffffffff81185d50&gt;] sys_open+0x20/0x30 00:34:43: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/3ce2436e-26aa-11e4-9de1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3ce2436e-26aa-11e4-9de1-5254006e85c2</a></p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/82/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/82/</a><br/> Distro/Arch: RHEL6.5/x86_64<br/> MDSCOUNT=2</p> <p>The same failure occurred: <a href="https://testing.hpdd.intel.com/test_sets/750a9454-28cf-11e4-85c7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/750a9454-28cf-11e4-85c7-5254006e85c2</a></p> <p>As per Oleg, this will be fixed in <a href="https://jira.whamcloud.com/browse/LU-5144" title="rename vs link deadlock" class="issue-link" data-issue-key="LU-5144"><del>LU-5144</del></a>.<br/> The patch for Lustre b2_5 branch is in <a href="http://review.whamcloud.com/10918" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10918</a>.</p> <p>an instance on master: <a href="https://testing.hpdd.intel.com/test_sets/49fc050a-44b7-11e4-bb5a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/49fc050a-44b7-11e4-bb5a-5254006e85c2</a></p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-5144" title="rename vs link deadlock" class="issue-link" data-issue-key="LU-5144"><del>LU-5144</del></a></p> Related LU-5144 Development Rank 1|hzw5jj: Rank (Obsolete) 11021 Severity