https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12404 Lustre <p>conf-sanity test_69 fails with 'create file after reformat'</p> <p>Looking at the client test log from the failure <a href="https://testing.whamcloud.com/test_sets/efc1357e-8895-11e9-8c65-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/efc1357e-8895-11e9-8c65-52540065bddc</a>, we see the following error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Starting client: trevis-18vm4.trevis.whamcloud.com: -o user_xattr,flock trevis-18vm11@tcp:/lustre /mnt/lustre CMD: trevis-18vm4.trevis.whamcloud.com mkdir -p /mnt/lustre CMD: trevis-18vm4.trevis.whamcloud.com mount -t lustre -o user_xattr,flock trevis-18vm11@tcp:/lustre /mnt/lustre touch: cannot touch '/mnt/lustre/d69.conf-sanity/f69.conf-sanity-last': No space left on device conf-sanity test_69: @@@@@@ FAIL: create file after reformat </pre> </div></div> <p>This looks like <a href="https://jira.whamcloud.com/browse/LU-8158" title="conf-sanity test_69: create file after reformat" class="issue-link" data-issue-key="LU-8158">LU-8158</a> but this is happening for non-SLES clients. </p> <p>Looking at the OST (vm6) console log, we see </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[37858.976026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [37859.170354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [37862.106450] LustreError: 30205:0:(ofd_dev.c:1709:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [37879.498553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_69: @@@@@@ FAIL: create file after reformat [37879.685068] Lustre: DEBUG MARKER: conf-sanity test_69: @@@@@@ FAIL: create file after reformat </pre> </div></div> <p>A different ofd_create_hdl() error is seen in <a href="https://jira.whamcloud.com/browse/LU-8158" title="conf-sanity test_69: create file after reformat" class="issue-link" data-issue-key="LU-8158">LU-8158</a>, but the root cause could be the same.</p> <p>We've started seeing this test fail with this ofd_create_hdl() error since 2019-05-27 Lustre version 2.12.53.62. Here are links to a few of the failed test session logs:<br/> <a href="https://testing.whamcloud.com/test_sets/e3585f0a-8148-11e9-a028-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e3585f0a-8148-11e9-a028-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/b1e108b8-8135-11e9-b8e0-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b1e108b8-8135-11e9-b8e0-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/5cf8cb50-83de-11e9-a028-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5cf8cb50-83de-11e9-a028-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/6bd12340-83d2-11e9-af1f-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6bd12340-83d2-11e9-af1f-52540065bddc</a></p> LU-12404

conf-sanity test 69 fails with 'create file after reformat'

Bug Minor Resolved Duplicate Sergey Cheremencev James Nunez Fri, 7 Jun 2019 18:46:12 +0000 Mon, 25 Nov 2019 20:59:32 +0000 Tue, 10 Sep 2019 18:30:06 +0000 Lustre 2.13.0 Lustre 2.12.3 0 3 <p>I suspect that this problem is caused by patch <a href="https://review.whamcloud.com/33833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33833</a> which was committed 2019-05-25 and affects <b>exactly</b> the number of objects created after reformat that test_69() is verifying:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit d07d9c5ed0aa1d6614944c7d1e0ca55cba301dc4 Author: Sergey Cheremencev &lt;c17829@cray.com&gt; AuthorDate: Fri Aug 24 17:03:45 2018 +0300 Commit: Oleg Drokin &lt;green@whamcloud.com&gt; CommitDate: Sat May 25 04:55:51 2019 +0000 LU-11760 ofd: formatted OST recognition change Modern system is fast enough to create above 100 000(5 * OST_MAX_PRECREATE) objects during commit interval. Increase the difference between MDS last_used ID and OST LAST_ID to 500 000 to avoid gaps after OST failover. </pre> </div></div> <p>The problem is that if the OST filesystem is does not have enough free inodes to store an extra 500k objects at recovery time, and the OST has previously created more objects than this, then the OST will run out of space during this test.</p> <p>I put a more detailed comment on how to fix this in <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a>. Maybe I should have left that ticket closed, and we should track the fix here?</p> <p>Suggest to leave this open. And do revert of <a href="https://review.whamcloud.com/#/c/33833/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33833/</a> with <a href="https://jira.whamcloud.com/browse/LU-12404" title="conf-sanity test 69 fails with &#39;create file after reformat&#39;" class="issue-link" data-issue-key="LU-12404"><del>LU-12404</del></a> while new patch will be landed under <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a>.</p> <p>Fixed under <a href="https://jira.whamcloud.com/browse/LU-11760" title="formatted OST recognition change" class="issue-link" data-issue-key="LU-11760"><del>LU-11760</del></a>.</p> Duplicate LU-12415 Related LU-8158 LU-11760 Development Rank 1|i00htr: Rank (Obsolete) 9223372036854775807 Severity