https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4427 Lustre <p>performance-sanity test_6 hung as follows on client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13:08:35:Lustre: DEBUG MARKER: ===== mdsrate-lookup-10dirs.sh Test preparation: creating 10 dirs with 26778 files. 13:08:35:INFO: task mdsrate:19365 blocked for more than 120 seconds. 13:08:35:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 13:08:35:mdsrate D 0000000000000001 0 19365 19361 0x00000080 13:08:35: ffff88005eba1ac8 0000000000000082 0000000000000000 ffff88006e406800 13:08:35: ffff88005eba1a88 ffffffffa0681629 ffff88007a0aee00 0000000000000003 13:08:35: ffff88006dfd2638 ffff88005eba1fd8 000000000000fb88 ffff88006dfd2638 13:08:35:Call Trace: 13:08:35: [&lt;ffffffffa0681629&gt;] ? __ptlrpc_request_bufs_pack+0x349/0x3c0 [ptlrpc] 13:08:35: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 13:08:35: [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 13:08:35: [&lt;ffffffffa084a8dc&gt;] mdc_reint+0x3c/0x3b0 [mdc] 13:08:35: [&lt;ffffffffa084beae&gt;] mdc_create+0x20e/0x780 [mdc] 13:08:35: [&lt;ffffffffa0ae6196&gt;] lmv_create+0x316/0x700 [lmv] 13:08:35: [&lt;ffffffffa09dbf46&gt;] ll_new_node+0x176/0x640 [lustre] 13:08:35: [&lt;ffffffffa09dc578&gt;] ll_mknod_generic+0x168/0x280 [lustre] 13:08:35: [&lt;ffffffff8118e0e3&gt;] ? generic_permission+0x23/0xb0 13:08:35: [&lt;ffffffffa09e2ef1&gt;] ll_create_nd+0x971/0xe80 [lustre] 13:08:35: [&lt;ffffffff8118e4c2&gt;] ? __lookup_hash+0x102/0x160 13:08:35: [&lt;ffffffff8118fbd4&gt;] vfs_create+0xb4/0xe0 13:08:35: [&lt;ffffffff81192800&gt;] sys_mknodat+0x280/0x2a0 13:08:35: [&lt;ffffffff8100bb8e&gt;] ? apic_timer_interrupt+0xe/0x20 13:08:35: [&lt;ffffffff8119283a&gt;] sys_mknod+0x1a/0x20 13:08:35: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/fc281616-73c4-11e3-b4ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fc281616-73c4-11e3-b4ff-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> FSTYPE=zfs<br/> LU-4427

performance-sanity test_6: mdsrate hung (sys_mknod)

Bug Blocker Resolved Fixed Nathaniel Clark Jian Yu zfs Fri, 3 Jan 2014 11:38:18 +0000 Mon, 21 Nov 2016 17:46:10 +0000 Mon, 21 Nov 2016 17:46:10 +0000 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.8.0 0 9 <p>By searching on Maloo, I found that performance-sanity passed on Lustre b2_5 build #3 (2.5.0) with FSTYPE=zfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/dc6ee504-37f4-11e3-8bc4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc6ee504-37f4-11e3-8bc4-52540035b04c</a></p> <p>A second run of performance-sanity test 6 with FSTYPE=zfs hit <a href="https://jira.whamcloud.com/browse/LU-3786" title="performance-sanity test_6: mkdir hung (sys_mkdir)" class="issue-link" data-issue-key="LU-3786"><del>LU-3786</del></a>.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> FSTYPE=zfs</p> <p>parallel-scale test statahead hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/74e5fee4-76b6-11e3-8c14-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/74e5fee4-76b6-11e3-8c14-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1)<br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/7a0a591c-a607-11e3-8a1b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7a0a591c-a607-11e3-8a1b-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/73/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/73/</a> (2.4.3 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p><a href="https://maloo.whamcloud.com/test_sets/e70e3c7c-ac60-11e3-81d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e70e3c7c-ac60-11e3-81d7-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_5/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/47/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/99e3e576-cb70-11e3-95c9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/99e3e576-cb70-11e3-95c9-52540035b04c</a></p> <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/61/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/61/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/50a6f7f0-ebe5-11e3-82b2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/50a6f7f0-ebe5-11e3-82b2-52540035b04c</a></p> <p>performance-sanity/6 hangs seems to fall into two catagories.<br/> 1) performance-sanity/5 fails (<a href="https://jira.whamcloud.com/browse/LU-4428" title="performance-sanity test_5: mknod(f59999) error: Disk quota exceeded" class="issue-link" data-issue-key="LU-4428">LU-4428</a> or <a href="https://jira.whamcloud.com/browse/LU-5146" title="Test failure performance-sanity test_5: readv failed: Connection reset by peer" class="issue-link" data-issue-key="LU-5146"><del>LU-5146</del></a>) then mdsrate hangs (this bug)<br/> 2) test_5 passes and mkdir hangs (<a href="https://jira.whamcloud.com/browse/LU-3786" title="performance-sanity test_6: mkdir hung (sys_mkdir)" class="issue-link" data-issue-key="LU-3786"><del>LU-3786</del></a>)</p> <p>FYI, by searching on Maloo, I found the failure did not occur on master branch. I also tried to reproduce the failure on master build #2820 but failed to hit the issue.</p> <p>Here are the test reports on master build #2820:</p> <p>performance-sanity:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b60591f8-9d6d-11e4-9d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b60591f8-9d6d-11e4-9d48-5254006e85c2</a> (ldiskfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/b915d632-9d6d-11e4-9d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b915d632-9d6d-11e4-9d48-5254006e85c2</a> (ldiskfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/34af27f2-9fe5-11e4-a43c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/34af27f2-9fe5-11e4-a43c-5254006e85c2</a> (zfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/98bf8674-9fe5-11e4-a43c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/98bf8674-9fe5-11e4-a43c-5254006e85c2</a> (zfs)</p> <p>parallel-scale:<br/> <a href="https://testing.hpdd.intel.com/test_sets/bc82d59a-9d6d-11e4-9d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bc82d59a-9d6d-11e4-9d48-5254006e85c2</a> (ldiskfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/c5ccc930-9d6d-11e4-9d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c5ccc930-9d6d-11e4-9d48-5254006e85c2</a> (ldiskfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/7af16a0c-9fc4-11e4-a450-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7af16a0c-9fc4-11e4-a450-5254006e85c2</a> (zfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/a98eeeb6-9fc4-11e4-a450-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a98eeeb6-9fc4-11e4-a450-5254006e85c2</a> (zfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/bffa1554-9fc4-11e4-a450-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bffa1554-9fc4-11e4-a450-5254006e85c2</a> (zfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/f702fb8e-9ef6-11e4-a23e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f702fb8e-9ef6-11e4-a23e-5254006e85c2</a> (zfs)<br/> <a href="https://testing.hpdd.intel.com/test_sets/fd7dc124-9ef6-11e4-a23e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fd7dc124-9ef6-11e4-a23e-5254006e85c2</a> (zfs)</p> <p>another instance: <br/> <a href="https://testing.hpdd.intel.com/test_sets/5ccde712-1251-11e5-bec9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ccde712-1251-11e5-bec9-5254006e85c2</a></p> <p>Does this still happen?</p> <blockquote><p>Does this still happen?</p></blockquote> <p>By searching on Maloo, I found the latest performance-sanity test 6 time out failures were all <a href="https://jira.whamcloud.com/browse/LU-3786" title="performance-sanity test_6: mkdir hung (sys_mkdir)" class="issue-link" data-issue-key="LU-3786"><del>LU-3786</del></a>.</p> <p>I can't find any problem in the report above. can it be that ZFS is just making progress too slow?</p> <p>The most recent failure of this type is from 2015-03-24 (all other perf-sanity/6 timeouts are <a href="https://jira.whamcloud.com/browse/LU-3786" title="performance-sanity test_6: mkdir hung (sys_mkdir)" class="issue-link" data-issue-key="LU-3786"><del>LU-3786</del></a>)<br/> <a href="https://testing.hpdd.intel.com/test_sets/d817106c-d325-11e4-a357-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d817106c-d325-11e4-a357-5254006e85c2</a></p> Duplicate LU-4408 Related LU-3786 Development Rank 1|hzwc4n: Rank (Obsolete) 12167 Severity