https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9908 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/c07d5048-8871-11e7-b93b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c07d5048-8871-11e7-b93b-5254006e85c2</a>.</p> <p>The sub-test test_41b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>test hangs &amp; fails during client umount of lustre. can't find root cause(s). have looked for OOPs or Panics with stack traces and can't find any.</p> <p>History search shows several similar fails on sles12sp2 recently.</p> <p>Info required for matching: conf-sanity 41b</p> LU-9908

conf-sanity test_41b: test failed to respond and timed out

Bug Minor Resolved Fixed Yang Sheng Maloo Thu, 24 Aug 2017 14:20:36 +0000 Tue, 24 Oct 2017 21:41:34 +0000 Mon, 16 Oct 2017 13:47:55 +0000 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.10.2 0 5 <p>this failure didn't reproduce on retest, so it's not a 100% fail.<br/> may still be a high rate fail on sles12 though.</p> <p>fails like this are happening in more places than test 41b.<br/> here's a similar looking fail in test 70e:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b3eda294-8905-11e7-b45f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b3eda294-8905-11e7-b45f-5254006e85c2</a></p> <p>once again it hangs during a client umount.<br/> can't find any panic or oops on any node.<br/> autotest times out an hour later and kills things.</p> <p>since conf-sanity on sles12sp2 is tested so little this failure may have been lurking for quite a long time.</p> <p>Looks like client hang:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:32:01:[15548.934823] Leftover inexact backtrace: 19:32:01:[15548.934823] 19:32:01:[15548.934826] umount S 0000000000000000 0 22377 22376 0x00000000 19:32:01:[15548.934827] ffff88007ae87a78 ffff8800641a1300 ffff88007c1b5800 ffff88007ae88000 19:32:01:[15548.934828] ffff88007ae87ab0 00000001003a2c78 ffff88007fc0e040 0000000000000000 19:32:01:[15548.934829] ffff88007ae87a90 ffffffff815e4c45 ffff88007fc0e040 ffff88007ae87b38 19:32:01:[15548.934830] Call Trace: 19:32:01:[15548.934832] [&lt;ffffffff815e4c45&gt;] schedule+0x35/0x80 19:32:01:[15548.934833] [&lt;ffffffff815e74d3&gt;] schedule_timeout+0x163/0x2d0 19:32:01:[15548.934857] [&lt;ffffffffa0994c7b&gt;] ptlrpc_set_wait+0x1cb/0x850 [ptlrpc] 19:32:01:[15548.934881] [&lt;ffffffffa0995378&gt;] ptlrpc_queue_wait+0x78/0x210 [ptlrpc] 19:32:01:[15548.934889] [&lt;ffffffffa0ac851b&gt;] mdc_statfs+0xab/0x2e0 [mdc] 19:32:01:[15548.934898] [&lt;ffffffffa092a1ce&gt;] lmv_statfs+0x26e/0xa30 [lmv] 19:32:01:[15548.934917] [&lt;ffffffffa0c3bbeb&gt;] ll_statfs_internal+0xeb/0xe00 [lustre] 19:32:01:[15548.934929] [&lt;ffffffffa0c3c97b&gt;] ll_statfs+0x7b/0x160 [lustre] 19:32:01:[15548.934932] [&lt;ffffffff8122dc13&gt;] statfs_by_dentry+0x93/0x110 19:32:01:[15548.934935] [&lt;ffffffff8122dca6&gt;] vfs_statfs+0x16/0xb0 19:32:01:[15548.934937] [&lt;ffffffff8122dd80&gt;] user_statfs+0x40/0x70 19:32:01:[15548.934939] [&lt;ffffffff8122ddc0&gt;] SYSC_statfs+0x10/0x30 19:32:01:[15548.934941] [&lt;ffffffff815e872e&gt;] entry_SYSCALL_64_fastpath+0x12/0x6d 19:32:01:[15548.936314] DWARF2 unwinder stuck at entry_SYSCALL_64_fastpath+0x12/0x6d 19:32:01:[15548.936314] </pre> </div></div> <p>this fail seems to be reproducing. Here's another one:<br/> <a href="https://testing.hpdd.intel.com/test_sets/515eed38-8c23-11e7-b94a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/515eed38-8c23-11e7-b94a-5254006e85c2</a></p> <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28767" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28767</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9908" title="conf-sanity test_41b: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9908"><del>LU-9908</del></a> tests: force umount client in test_70e<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 77ef8e40b0457fac025475b99fb898dd5904a2fd</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8ec26fc2-935a-11e7-b74a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8ec26fc2-935a-11e7-b74a-5254006e85c2</a></p> <p>the patch <a href="https://review.whamcloud.com/28767" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28767</a> changes (fixes?) test 70e, but does nothing for similar fails seen in test 41b.</p> <p>Here's another seen on b2_10 in 41b:<br/> <a href="https://testing.hpdd.intel.com/test_sets/dc983726-98dd-11e7-ba20-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dc983726-98dd-11e7-ba20-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d2071558-9ca6-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d2071558-9ca6-11e7-b778-5254006e85c2</a></p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29108" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29108</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9908" title="conf-sanity test_41b: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9908"><del>LU-9908</del></a> tests: force umount client in test_70e &amp; 41b<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 8d3a564a82a2cdef304638b339886f2de991bdca</p> <p>another one on b2_10 branch 2.10.1 RC1 testing with SLES12sp2 client<br/> <a href="https://testing.hpdd.intel.com/test_sets/eaf41ec6-9c1f-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/eaf41ec6-9c1f-11e7-b778-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28767/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28767/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9908" title="conf-sanity test_41b: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9908"><del>LU-9908</del></a> tests: force umount client in test 70e, 41b, and 105<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 20787a89ad7dc99290c4d9cff8247b0a532e92b9</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29108/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29108/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9908" title="conf-sanity test_41b: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9908"><del>LU-9908</del></a> tests: force umount client in test 70e, 41b, and 105<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 1361b447bb68757f6f41ce7d5cbec954c727c31d</p> Blocker LU-9469 Development Rank 1|hzzixr: Rank (Obsolete) 9223372036854775807 Severity