https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3786 Lustre <p>performance-sanity test_6 hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:15:46:Lustre: DEBUG MARKER: ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 6904 files. 06:15:46:INFO: task mkdir:3833 blocked for more than 120 seconds. 06:15:46:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 06:15:46:mkdir D 0000000000000000 0 3833 3826 0x00000080 06:15:46: ffff88000e523dd8 0000000000000086 ffffffffa10ccf38 0000000000000000 06:15:46: ffff88000e523e48 ffff88007afe7540 ffff88007afe7540 ffff88007afe7540 06:15:46: ffff88007afe7af8 ffff88000e523fd8 000000000000fb88 ffff88007afe7af8 06:15:46:Call Trace: 06:15:46: [&lt;ffffffff811a1c00&gt;] ? mntput_no_expire+0x30/0x110 06:15:46: [&lt;ffffffff8150f3de&gt;] __mutex_lock_slowpath+0x13e/0x180 06:15:46: [&lt;ffffffff8150f27b&gt;] mutex_lock+0x2b/0x50 06:15:46: [&lt;ffffffff8118e930&gt;] lookup_create+0x30/0xd0 06:17:49: [&lt;ffffffff8119221c&gt;] sys_mkdirat+0x7c/0x130 06:17:49: [&lt;ffffffff811a1c00&gt;] ? mntput_no_expire+0x30/0x110 06:17:49: [&lt;ffffffff810dc867&gt;] ? audit_syscall_entry+0x1d7/0x200 06:17:49: [&lt;ffffffff811922e8&gt;] sys_mkdir+0x18/0x20 06:17:49: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 06:17:50:INFO: task mkdir:3835 blocked for more than 120 seconds. </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/8358068c-0962-11e3-ad8a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8358068c-0962-11e3-ad8a-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/33/">http://build.whamcloud.com/job/lustre-b2_4/33/</a><br/> FSTYPE=zfs<br/> LU-3786

performance-sanity test_6: mkdir hung (sys_mkdir)

Bug Major Resolved Cannot Reproduce Nathaniel Clark Jian Yu 22pl zfs Tue, 20 Aug 2013 14:58:16 +0000 Wed, 22 Mar 2023 23:56:56 +0000 Fri, 21 May 2021 18:43:42 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.5.3 Lustre 2.8.0 Lustre 2.10.0 0 10 <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/29/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/29/</a><br/> FSTYPE=zfs</p> <p>The same failure: <a href="https://maloo.whamcloud.com/test_sets/20ed20e6-042a-11e3-a8e9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/20ed20e6-042a-11e3-a8e9-52540035b04c</a></p> <p>The autotest logs looks like mpi or something in the test is not configured correctly? </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found -------------------------------------------------------------------------- [[49900,1],6]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: wtm-15vm5.rosso.whamcloud.com Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found librdmacm: Fatal: no RDMA devices found [wtm-15vm6.rosso.whamcloud.com:03820] 9 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics [wtm-15vm6.rosso.whamcloud.com:03820] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages </pre> </div></div> <p>Also the timestamps don't line up well on the Client</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:15:46:Lustre: DEBUG MARKER: /usr/sbin/lctl mark ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 6904 files. 06:15:46:Lustre: DEBUG MARKER: ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 6904 files. 06:15:46:INFO: task mkdir:3833 blocked for more than 120 seconds. &lt;========= How has this task ran for more than 120 seconds? It just started? 06:15:46:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 06:15:46:mkdir D 0000000000000000 0 3833 3826 0x00000080 06:15:46: ffff88000e523dd8 0000000000000086 ffffffffa10ccf38 0000000000000000 06:15:46: ffff88000e523e48 ffff88007afe7540 ffff88007afe7540 ffff88007afe7540 06:15:46: ffff88007afe7af8 ffff88000e523fd8 000000000000fb88 ffff88007afe7af8 06:15:46:Call Trace: 06:15:46: [&lt;ffffffff811a1c00&gt;] ? mntput_no_expire+0x30/0x110 06:15:46: [&lt;ffffffff8150f3de&gt;] __mutex_lock_slowpath+0x13e/0x180 06:15:46: [&lt;ffffffff8150f27b&gt;] mutex_lock+0x2b/0x50 06:15:46: [&lt;ffffffff8118e930&gt;] lookup_create+0x30/0xd0 06:17:49: [&lt;ffffffff8119221c&gt;] sys_mkdirat+0x7c/0x130 &lt;==== Random 2 min jump </pre> </div></div> <p>This might be helped by Alex's patch <a href="http://review.whamcloud.com/7343" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7343</a>.</p> <p>Keith, it might depend on where the timestamps are generated (e.g. stuck node or console logger) and/or when the console output is being sent. The "Random 2 min jump" lines up pretty well with "blocked for more than 120 seconds"...</p> <p>We are seeing alot of this on Rosso. I would expect to the 2 min gap between starting the test and the "blocked for more than 120 seconds message" not just during the printing of the backtrace. There is an open TT TT-1615 to look into this issue sometimes it is more than 2 min... Maybe it is just a weird console timestamp. </p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/46/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/46/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/b72e57a6-3061-11e3-b28a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b72e57a6-3061-11e3-b28a-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> OSTCOUNT=2</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/fcce24ee-4452-11e3-8472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fcce24ee-4452-11e3-8472-52540035b04c</a></p> <p>Seen only on zfs and only on full tests and only on b2_4.<br/> Only 5 instances seen total.</p> <p>Think full tests is a red herring. Suspect performance-sanity is only run in full tests, not in other groups of tests. As far as I can tell all instances of both PASS and TIMEOUT are only in full tests.</p> <p>The same failure also occurred on master branch with FSTYPE=zfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/0951f362-4f2f-11e3-9462-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0951f362-4f2f-11e3-9462-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/50/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/50/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/7bb7ce82-4c9f-11e3-8ab0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7bb7ce82-4c9f-11e3-8ab0-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-reviews/19580/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/19580/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs<br/> MDSCOUNT=4</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/0951f362-4f2f-11e3-9462-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0951f362-4f2f-11e3-9462-52540035b04c</a><br/> Console log on Client 2 (wtm-25vm6):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:08:01:Lustre: DEBUG MARKER: ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 6793 files. 09:08:01:INFO: task mkdir:17358 blocked for more than 120 seconds. 09:08:01:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 09:08:01:mkdir D 0000000000000000 0 17358 17349 0x00000080 09:08:01: ffff88006e74fdd8 0000000000000082 ffffffffa0ab7260 0000000000000000 09:08:01: ffff88006e74fe48 ffff880037bb9540 ffff880037bb9540 ffff880037bb9540 09:08:01: ffff880037bb9af8 ffff88006e74ffd8 000000000000fb88 ffff880037bb9af8 09:08:01:Call Trace: 09:08:01: [&lt;ffffffff811a2100&gt;] ? mntput_no_expire+0x30/0x110 09:08:01: [&lt;ffffffff8150f81e&gt;] __mutex_lock_slowpath+0x13e/0x180 09:08:01: [&lt;ffffffff8150f6bb&gt;] mutex_lock+0x2b/0x50 09:08:01: [&lt;ffffffff8118ebe0&gt;] lookup_create+0x30/0xd0 09:08:01: [&lt;ffffffff811924dc&gt;] sys_mkdirat+0x7c/0x130 09:08:01: [&lt;ffffffff811a2100&gt;] ? mntput_no_expire+0x30/0x110 09:08:01: [&lt;ffffffff810dc937&gt;] ? audit_syscall_entry+0x1d7/0x200 09:08:01: [&lt;ffffffff811925a8&gt;] sys_mkdir+0x18/0x20 09:08:01: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/67/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/67/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/9489aa9a-65d9-11e3-9f05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9489aa9a-65d9-11e3-9f05-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/56a48888-76b6-11e3-8c14-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/56a48888-76b6-11e3-8c14-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/40/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/40/</a> (2.5.1 RC2)<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/77cf0a5a-a84e-11e3-98cc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/77cf0a5a-a84e-11e3-98cc-52540035b04c</a></p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/84/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/84/</a><br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d0672d9c-2b24-11e4-bb80-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d0672d9c-2b24-11e4-bb80-5254006e85c2</a></p> <p>lustre-master build #3071</p> <p><a href="https://testing.hpdd.intel.com/test_sets/925acf00-13e4-11e5-9804-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/925acf00-13e4-11e5-9804-5254006e85c2</a></p> <p>Another instance for EL6.7 Server/EL6.7 Client - ZFS<br/> Master, build# 3270<br/> <a href="https://testing.hpdd.intel.com/test_sets/99d7b9ee-a275-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/99d7b9ee-a275-11e5-bdef-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9f5fa944-a275-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f5fa944-a275-11e5-bdef-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/9abecea0-bc84-11e5-b3b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9abecea0-bc84-11e5-b3b7-5254006e85c2</a><br/> lustre-master build# 3305 RHEL6.7 zfs</p> <p>Another instance for FULL - EL6.7 Server/EL6.7 Client - ZFS , master, build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/a60b5630-cb47-11e5-a59a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a60b5630-cb47-11e5-a59a-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/a60b5630-cb47-11e5-a59a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a60b5630-cb47-11e5-a59a-5254006e85c2</a></p> <p>also seen on master tag-2.9.53 RHEL7.3 zfs<br/> <a href="https://testing.hpdd.intel.com/test_sets/18a65a20-f8f9-11e6-aac4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/18a65a20-f8f9-11e6-aac4-5254006e85c2</a></p> Related LU-16658 LU-3753 LU-4427 Development Rank 1|hzvym7: Rank (Obsolete) 9793 Severity