https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11152 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/28765bb2-8831-11e8-9e83-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/28765bb2-8831-11e8-9e83-52540065bddc</a></p> <p>test_133g failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$'ost1 find /proc/fs/lustre/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed' </pre> </div></div> <p>This seems to be only failing on <tt>review-dne-zfs-part-1</tt> and <tt>review-dne-part-1</tt>.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_133g - $'ost1 find /proc/fs/lustre/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed'</p> LU-11152

sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed

Bug Minor Resolved Fixed Andreas Dilger Maloo Tue, 17 Jul 2018 18:27:51 +0000 Thu, 19 Dec 2019 00:17:11 +0000 Thu, 19 Dec 2019 00:16:29 +0000 Lustre 2.12.0 Lustre 2.14.0 0 5 <p>This is similar to <a href="https://jira.whamcloud.com/browse/LU-9700" title="Interop 2.9.0&lt;-&gt;master sanity test_133f: proc file read failed" class="issue-link" data-issue-key="LU-9700">LU-9700</a>, but no interop involved. <a href="https://jira.whamcloud.com/browse/LU-10038" title="sanity test 133g fails with “ &#39;$&#39;mds1 find /proc/fs/lustre/n/proc/sys/lnet/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed&#39;&#39; " class="issue-link" data-issue-key="LU-10038"><del>LU-10038</del></a> was reported fixed for 2.11, but this problem is still being seen (about once a day in the past two weeks).</p> <p>Also seeing some cases like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0000/exports/10.9.1.249@tcp’: No such file or directory trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0005/exports/10.9.1.249@tcp’: No such file or directory trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0006/exports/10.9.1.249@tcp’: No such file or directory trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0007/exports/10.9.1.249@tcp’: No such file or directory </pre> </div></div> <p>This is a normal race which we handle by passing the <tt>-ignore_readdir_race</tt> flag to find. See <a href="https://jira.whamcloud.com/browse/LU-10038" title="sanity test 133g fails with “ &#39;$&#39;mds1 find /proc/fs/lustre/n/proc/sys/lnet/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed&#39;&#39; " class="issue-link" data-issue-key="LU-10038"><del>LU-10038</del></a>. However, in find 4.5.11 which is included in CentOS 7.3-7.5 the <tt>-ignore_readdir_race</tt> flag is ineffective. See <a href="https://bugs.centos.org/view.php?id=13685" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.centos.org/view.php?id=13685</a>.</p> <p>John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32934" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32934</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> test: work around find bug in sanity 133<span class="error">&#91;fg&#93;</span><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aa345cb2e7eeba0699157a1a16e242e9a1a54a4e</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32934/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32934/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> test: work around find bug in sanity 133<span class="error">&#91;fg&#93;</span><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: edd1b078941b9cc97e39e0b54b9daabe0e6f2792</p> <p>Landed for 2.12</p> <p><del>James Simmons (uja.ornl@yahoo.com) uploaded a new patch:</del><br/> <a href="https://review.whamcloud.com/33408" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33408</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: correctly cleanup the FMR/FastReg pools</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 5ea8405a7be061bd485c2f94c09d14ccfbac48b0</del></p> <p><del>Oleg Drokin (green@whamcloud.com) merged in patch</del><br/> <a href="https://review.whamcloud.com/33408/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33408/</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: test fpo_fmr_poool pointer instead of special bool</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set:</del><br/> <del>Commit: 9b790ba0f5606c0a91563828fa43f5e4ae210425</del></p> <p><del>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch:</del><br/> <a href="https://review.whamcloud.com/33802" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33802</a><br/> <del>Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: test fpo_fmr_poool pointer instead of special bool"</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: b169980b805b78df9b05e10079d27c4a6d3dbbc1</del></p> <p><del>Oleg Drokin (green@whamcloud.com) merged in patch</del><br/> <a href="https://review.whamcloud.com/33802/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33802/</a><br/> <del>Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: test fpo_fmr_poool pointer instead of special bool"</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set:</del><br/> <del>Commit: a65d072fa45fc90c2fc74b61d214de79c0bf33e5</del></p> <p>Reverting because this has introduced <a href="https://jira.whamcloud.com/browse/LU-11735" title="o2iblnd - bad check for fmr" class="issue-link" data-issue-key="LU-11735">LU-11735</a></p> <p>It seems that we have a bit of mix-up here.</p> <p>9b790ba0f5606c0a91563828fa43f5e4ae210425 (<a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: test fpo_fmr_poool pointer instead of special bool) was landed for "<a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a>", but it was really intended for <a href="https://jira.whamcloud.com/browse/LU-11552" title="improper FMR/FastReg pool cleanup" class="issue-link" data-issue-key="LU-11552">LU-11552</a>. The FMR pool stuff shouldn't have anything to do with this ticket.</p> <p>Then it was reverted with:<br/> commit a65d072fa45fc90c2fc74b61d214de79c0bf33e5<br/> Author: Amir Shehata &lt;ashehata@whamcloud.com&gt;<br/> Date: Thu Dec 6 20:52:22 2018 +0000</p> <p> Revert "<a href="https://jira.whamcloud.com/browse/LU-11152" title="sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed" class="issue-link" data-issue-key="LU-11152"><del>LU-11152</del></a> lnd: test fpo_fmr_poool pointer instead of special bool"</p> <p> This reverts commit 9b790ba0f5606c0a91563828fa43f5e4ae210425.</p> <p>And then this ticket was implicated.</p> <p>I don't know how to clean up this mess, but I'm going to post a duplicate comment in <a href="https://jira.whamcloud.com/browse/LU-11552" title="improper FMR/FastReg pool cleanup" class="issue-link" data-issue-key="LU-11552">LU-11552</a>. Maybe we don't need to do anything special other than to add a helpful comment to the commit message if/when <a href="https://jira.whamcloud.com/browse/LU-11735" title="o2iblnd - bad check for fmr" class="issue-link" data-issue-key="LU-11735">LU-11735</a> or <a href="https://jira.whamcloud.com/browse/LU-11552" title="improper FMR/FastReg pool cleanup" class="issue-link" data-issue-key="LU-11552">LU-11552</a> are addressed.</p> <p>Cory, thanks for pointing out the discrepancy here. Too bad this wasn't caught before these patches were landed, but it isn't possible to change the commit comments in Git.</p> Related LU-9700 LU-10038 LU-11552 LU-11735 Development Rank 1|hzzz7b: Rank (Obsolete) 9223372036854775807 Severity