https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6088 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/9ec8122a-9608-11e4-af28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ec8122a-9608-11e4-af28-5254006e85c2</a>.</p> <p>The sub-test test_1 failed with the following logs on the client console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>INFO: task dir_create.sh:5686 was blocked for more than 120s Call trace: mutex_lock+0x2b/0x50 do_lookup+0x11b/0x230 __link_path_walk+0x200/0x1000 path_walk+0x6a/0xe0 do_filp_open+0x1fa/0xd20 do_sys_open+0x69/0x140 sys_open+0x20/0x30 </pre> </div></div> <p>It looks like this is only being hit with both master client and master server (pre-2.7.0) so is very likely related to DNE striped directories and is a regression on master (possibly due to the addition of a new racer test for striped directories?). Combinations of 2.4/2.5/2.6/master client or server do not hit this problem. </p> <p>It would be nice to get the <a href="https://jira.whamcloud.com/browse/LU-4712" title="racer test_1: oops at __d_lookup+0x8c" class="issue-link" data-issue-key="LU-4712"><del>LU-4712</del></a> patch <a href="http://review.whamcloud.com/9689" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9689</a> landed to clean up the DNE striped directory console messages, but this case doesn't have the client oops, just stuck threads.</p> <p>Info required for matching: racer 1</p> LU-6088

racer test_1: dir_create.sh mutex deadlock in sys_open->do_lookup

Bug Critical Resolved Fixed Di Wang Maloo HB Wed, 7 Jan 2015 19:10:11 +0000 Tue, 2 May 2023 07:51:56 +0000 Fri, 23 Jan 2015 05:21:24 +0000 Lustre 2.7.0 Lustre 2.7.0 0 5 <p>The same stack trace has been seen in <a href="https://jira.whamcloud.com/browse/LU-6085" title="racer stuck on mutex_lock in ll_setattr_raw()" class="issue-link" data-issue-key="LU-6085"><del>LU-6085</del></a></p> <p>Hmm, this trace caused my attention, </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ls S 0000000000000001 0 33694 1 0x00000080 ffff8801d21fd2f8 0000000000000082 ffff8801d21fd288 ffffffffa092ac0c ffffffffa0a0b460 ffff8801e5393000 ffff8801d21fd268 ffffffffa093d4f5 ffff8801d21fd2f8 ffffffffa0964af2 ffff8801e5ed25f8 ffff8801d21fdfd8 Call Trace: [&lt;ffffffffa092ac0c&gt;] ? ptlrpc_request_bufs_pack+0x5c/0x80 [ptlrpc] [&lt;ffffffffa093d4f5&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] [&lt;ffffffffa0964af2&gt;] ? __req_capsule_get+0x162/0x6d0 [ptlrpc] [&lt;ffffffffa0941d40&gt;] ? lustre_swab_mdt_body+0x0/0x140 [ptlrpc] [&lt;ffffffffa06e8fe4&gt;] obd_get_request_slot+0x1a4/0x280 [obdclass] [&lt;ffffffff81064b90&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0ba11a5&gt;] mdc_enqueue+0x275/0x1a40 [mdc] [&lt;ffffffffa0b9f25b&gt;] ? mdc_lock_match+0xbb/0x170 [mdc] [&lt;ffffffffa0ba2b52&gt;] mdc_intent_lock+0x1e2/0x5f9 [mdc] [&lt;ffffffffa1174af0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa0912840&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa0b59b32&gt;] lmv_revalidate_slaves+0x482/0x1130 [lmv] [&lt;ffffffffa1174af0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa0b40a7a&gt;] lmv_update_lsm_md+0x1a/0x20 [lmv] [&lt;ffffffffa11562da&gt;] ll_update_inode+0x134a/0x1e60 [lustre] [&lt;ffffffffa0b5c3d1&gt;] ? lmv_fld_lookup+0xf1/0x440 [lmv] [&lt;ffffffff8129456a&gt;] ? strlcpy+0x4a/0x60 [&lt;ffffffffa1156e78&gt;] ll_read_inode2+0x88/0x470 [lustre] [&lt;ffffffffa11720fb&gt;] ll_iget+0x13b/0x3c0 [lustre] [&lt;ffffffffa0b3e4b8&gt;] ? lmv_get_lustre_md+0x88/0x300 [lmv] [&lt;ffffffffa1164fe5&gt;] ll_prep_inode+0x6c5/0xe80 [lustre] [&lt;ffffffffa0929c4f&gt;] ? ptlrpc_request_cache_free+0xbf/0x100 [ptlrpc] [&lt;ffffffffa0b59064&gt;] ? lmv_intent_remote+0x444/0xa90 [lmv] [&lt;ffffffffa0941d40&gt;] ? lustre_swab_mdt_body+0x0/0x140 [ptlrpc] [&lt;ffffffffa11755d1&gt;] ll_lookup_it_finish+0x2f1/0x11b0 [lustre] [&lt;ffffffff811749e3&gt;] ? kmem_cache_alloc_trace+0x1a3/0x1b0 [&lt;ffffffffa1171ed9&gt;] ? ll_i2suppgid+0x19/0x30 [lustre] [&lt;ffffffffa115748c&gt;] ? ll_prep_md_op_data+0x22c/0x530 [lustre] [&lt;ffffffffa1174af0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa1176737&gt;] ll_lookup_it+0x2a7/0x9a0 [lustre] [&lt;ffffffffa1176eb9&gt;] ll_lookup_nd+0x89/0x5e0 [lustre] [&lt;ffffffff8119e0f5&gt;] do_lookup+0x1a5/0x230 [&lt;ffffffff8119ed84&gt;] __link_path_walk+0x7a4/0x1000 [&lt;ffffffff8114f89f&gt;] ? handle_pte_fault+0x4af/0xb00 [&lt;ffffffff8119f89a&gt;] path_walk+0x6a/0xe0 [&lt;ffffffff8119faab&gt;] filename_lookup+0x6b/0xc0 [&lt;ffffffff8122db26&gt;] ? security_file_alloc+0x16/0x20 [&lt;ffffffff811a0f84&gt;] do_filp_open+0x104/0xd20 [&lt;ffffffff8129980a&gt;] ? strncpy_from_user+0x4a/0x90 [&lt;ffffffff811ae432&gt;] ? alloc_fd+0x92/0x160 [&lt;ffffffff8118b237&gt;] do_sys_open+0x67/0x130 [&lt;ffffffff8118b340&gt;] sys_open+0x20/0x30 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Right now, when the slave lock is being revalidated (enqueue etc), we do not release the master lock, it is not probably not right. I will cook a patch. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13432" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13432</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6088" title="racer test_1: dir_create.sh mutex deadlock in sys_open-&gt;do_lookup" class="issue-link" data-issue-key="LU-6088"><del>LU-6088</del></a> lmv: Do not revalidate strips with master lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 715252971aa4839930492520a72eb50b4fa2936b</p> <p>With this patch and minor change for racer, racer can pass on my local test (4 MDTs, 2 OSTs). </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/tests/cfg/local.sh b/lustre/tests/cfg/local.sh index 6d16312..73c85a3 100644 --- a/lustre/tests/cfg/local.sh +++ b/lustre/tests/cfg/local.sh @@ -13,7 +13,7 @@ TMP=${TMP:-/tmp} DAEMONSIZE=${DAEMONSIZE:-500} MDSCOUNT=${MDSCOUNT:-1} MDSDEVBASE=${MDSDEVBASE:-$TMP/${FSNAME}-mdt} -MDSSIZE=${MDSSIZE:-200000} +MDSSIZE=${MDSSIZE:-2000000} # # Format options of facets can be specified with these variables: # @@ -39,7 +39,7 @@ MGS_MOUNT_OPTS=${MGS_MOUNT_OPTS:-} OSTCOUNT=${OSTCOUNT:-2} OSTDEVBASE=${OSTDEVBASE:-$TMP/${FSNAME}-ost} -OSTSIZE=${OSTSIZE:-200000} +OSTSIZE=${OSTSIZE:-2000000} OSTOPT=${OSTOPT:-} OST_FS_MKFS_OPTS=${OST_FS_MKFS_OPTS:-} OST_MOUNT_OPTS=${OST_MOUNT_OPTS:-} diff --git a/lustre/tests/racer/racer.sh b/lustre/tests/racer/racer.sh index deef18e..3ed624e 100755 --- a/lustre/tests/racer/racer.sh +++ b/lustre/tests/racer/racer.sh @@ -17,7 +17,7 @@ file_list file_concat file_exec file_chown file_chmod file_mknod file_truncate \ file_delxattr file_getxattr file_setxattr" if [ $MDSCOUNT -gt 1 ]; then - RACER_PROGS="${RACER_PROGS} dir_remote dir_migrate" + RACER_PROGS="${RACER_PROGS} dir_remote" fi racer_cleanup() </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== racer test 1: racer on clients: testnode DURATION=300 == 17:12:47 (1421284367) racers pids: 77271 77272 77273 77275 77278 77281 77285 77289 ./file_exec.sh: line 12: 86522 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 90169 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 95671 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 95382 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 98325 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 108967 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 115776 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 122252 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 122042 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 130326 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 12459 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 16796 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 36610 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 40410 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 41512 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 44574 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 54088 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 53171 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 55728 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 56052 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 58816 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 65907 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 68804 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 74136 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 73959 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 78464 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 93150 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 98385 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 102325 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 101154 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 104116 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 106750 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 113145 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 113147 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 113766 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 115832 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 117893 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 122696 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 421 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 9260 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 11327 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 11681 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 17643 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 20928 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 24589 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 38846 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 49343 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 64789 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 64736 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 89863 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 103920 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 105115 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 107463 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 116236 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 116222 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 117338 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 125041 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 126624 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 5690 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 10494 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 21195 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 21185 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 28111 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 30436 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 37965 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 39235 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 42352 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 45837 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 46220 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 47790 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 52719 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 53661 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 53811 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 54609 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 55437 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 60614 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 65795 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 66303 Segmentation fault $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 71790 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 74574 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 80161 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 86993 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 86851 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 86691 Segmentation fault (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 90074 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 91221 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 88688 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 104944 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 110164 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 115085 Bus error $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null ./file_exec.sh: line 12: 121941 Bus error (core dumped) $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/null file_create.sh: no process killed dir_create.sh: no process killed file_rm.sh: no process killed file_create.sh: no process killed file_create.sh: no process killed dir_create.sh: no process killed file_rename.sh: no process killed dir_create.sh: no process killed file_rm.sh: no process killed file_link.sh: no process killed file_rm.sh: no process killed file_rename.sh: no process killed file_symlink.sh: no process killed file_create.sh: no process killed file_rename.sh: no process killed file_link.sh: no process killed file_create.sh: no process killed file_list.sh: no process killed file_create.sh: no process killed dir_create.sh: no process killed file_link.sh: no process killed file_symlink.sh: no process killed dir_create.sh: no process killed file_concat.sh: no process killed file_create.sh: no process killed file_symlink.sh: no process killed file_rm.sh: no process killed file_rm.sh: no process killed dir_create.sh: no process killed file_list.sh: no process killed file_create.sh: no process killed file_exec.sh: no process killed dir_create.sh: no process killed file_list.sh: no process killed file_rename.sh: no process killed file_concat.sh: no process killed file_rm.sh: no process killed file_rename.sh: no process killed file_chown.sh: no process killed dir_create.sh: no process killed file_rm.sh: no process killed file_concat.sh: no process killed file_link.sh: no process killed file_exec.sh: no process killed file_rename.sh: no process killed file_link.sh: no process killed file_chmod.sh: no process killed file_rm.sh: no process killed file_rename.sh: no process killed file_exec.sh: no process killed file_symlink.sh: no process killed file_mknod.sh: no process killed file_symlink.sh: no process killed file_link.sh: no process killed file_rename.sh: no process killed file_chown.sh: no process killed file_link.sh: no process killed file_chown.sh: no process killed file_truncate.sh: no process killed file_list.sh: no process killed file_list.sh: no process killed file_symlink.sh: no process killed file_link.sh: no process killed file_symlink.sh: no process killed file_chmod.sh: no process killed file_delxattr.sh: no process killed file_chmod.sh: no process killed file_concat.sh: no process killed file_list.sh: no process killed file_list.sh: no process killed file_concat.sh: no process killed file_symlink.sh: no process killed file_getxattr.sh: no process killed file_mknod.sh: no process killed file_mknod.sh: no process killed file_exec.sh: no process killed file_concat.sh: no process killed file_concat.sh: no process killed file_list.sh: no process killed file_exec.sh: no process killed file_setxattr.sh: no process killed file_truncate.sh: no process killed file_truncate.sh: no process killed file_chown.sh: no process killed file_exec.sh: no process killed file_delxattr.sh: no process killed file_concat.sh: no process killed file_chown.sh: no process killed file_exec.sh: no process killed file_delxattr.sh: no process killed file_chown.sh: no process killed dir_remote.sh: no process killed file_chmod.sh: no process killed file_chmod.sh: no process killed file_exec.sh: no process killed file_getxattr.sh: no process killed file_getxattr.sh: no process killed file_chown.sh: no process killed file_chmod.sh: no process killed file_mknod.sh: no process killed file_setxattr.sh: no process killed file_mknod.sh: no process killed file_setxattr.sh: no process killed file_mknod.sh: no process killed file_chown.sh: no process killed file_chmod.sh: no process killed file_truncate.sh: no process killed dir_remote.sh: no process killed dir_remote.sh: no process killed file_truncate.sh: no process killed file_truncate.sh: no process killed file_chmod.sh: no process killed file_mknod.sh: no process killed file_delxattr.sh: no process killed file_delxattr.sh: no process killed file_delxattr.sh: no process killed file_getxattr.sh: no process killed file_mknod.sh: no process killed file_getxattr.sh: no process killed file_getxattr.sh: no process killed file_setxattr.sh: no process killed file_truncate.sh: no process killed Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre2 We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre2 We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. file_truncate.sh: no process killed file_setxattr.sh: no process killed file_setxattr.sh: no process killed file_delxattr.sh: no process killed dir_remote.sh: no process killed dir_remote.sh: no process killed Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre2 We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. dir_remote.sh: no process killed file_delxattr.sh: no process killed file_getxattr.sh: no process killed file_getxattr.sh: no process killed file_setxattr.sh: no process killed file_setxattr.sh: no process killed dir_remote.sh: no process killed dir_remote.sh: no process killed Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. pid=77271 rc=0 Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. pid=77272 rc=0 Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre2 We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. Running /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on testnode@tcp:/lustre 3777312 143096 3417764 5% /mnt/lustre We survived /work/lustre-release_new/lustre/tests/racer/racer.sh for 300 seconds. pid=77273 rc=0 pid=77275 rc=0 pid=77278 rc=0 pid=77281 rc=0 pid=77285 rc=0 pid=77289 rc=0 Resetting fail_loc on all nodes...done. PASS 1 (306s) == racer test complete, duration 308 sec == 17:17:53 (1421284673) Stopping clients: testnode /mnt/lustre2 (opts:) Stopping client testnode /mnt/lustre2 opts: [root@testnode tests]# [root@testnode tests]# MDSCOUNT=4 sh llmountcleanup.sh Stopping clients: testnode /mnt/lustre (opts:-f) Stopping client testnode /mnt/lustre opts:-f Stopping clients: testnode /mnt/lustre2 (opts:-f) Stopping /mnt/mds1 (opts:-f) on testnode Stopping /mnt/mds2 (opts:-f) on testnode Stopping /mnt/mds3 (opts:-f) on testnode Stopping /mnt/mds4 (opts:-f) on testnode Stopping /mnt/ost1 (opts:-f) on testnode Stopping /mnt/ost2 (opts:-f) on testnode modules unloaded. </pre> </div></div> <p>Note: I run racer with 8 cores and 8G memory.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13432/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13432/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6088" title="racer test_1: dir_create.sh mutex deadlock in sys_open-&gt;do_lookup" class="issue-link" data-issue-key="LU-6088"><del>LU-6088</del></a> lmv: Do not revalidate stripes with master lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0860eda0544a507754b3af4aadcd651e1120ded5</p> <p>Landed for 2.7</p> Duplicate LU-5936 Related LU-6085 LU-4712 LU-5285 Development Rank 1|hzx3d3: Rank (Obsolete) 16947 Severity