Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-10059] sanityn test_32a: wrong file size https://jira.whamcloud.com/browse/LU-10059 Lustre <p>This issue was created by maloo for Bob Glossman <bob.glossman@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/12f0d24e-a732-11e7-b786-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/12f0d24e-a732-11e7-b786-5254006e85c2</a>.</p> <p>The sub-test test_32a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>wrong file size </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanityn 32a</p> LU-10059

sanityn test_32a: wrong file size

Bug Minor Resolved Fixed WC Triage Maloo DNE Mon, 2 Oct 2017 20:37:47 +0000 Mon, 16 May 2022 21:51:26 +0000 Thu, 17 Mar 2022 16:06:33 +0000 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.2 Lustre 2.14.0 Lustre 2.12.6 1 11 <p>More failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/937fd378-b3f4-11e7-a282-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/937fd378-b3f4-11e7-a282-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4b6b05f2-cd8b-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4b6b05f2-cd8b-11e7-a066-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/7fef1724-db22-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7fef1724-db22-11e7-9c63-52540065bddc</a></p> <p>+1 on master<br/> <a href="https://testing.hpdd.intel.com/test_sets/45ffb738-569e-11e8-b303-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/45ffb738-569e-11e8-b303-52540065bddc</a></p> <p>This test does not disable lockless truncate on failure, which is not good, since it's not generally safe. I realized this from:<br/> <a href="https://testing.whamcloud.com/test_sets/b5edcc08-1b7c-11e9-8388-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b5edcc08-1b7c-11e9-8388-52540065bddc</a></p> <p>Which panicked with an error on truncate.</p> <p>From 32a:<br/> " log "checking cached lockless truncate"<br/> $TRUNCATE $DIR1/$tfile 8000000<br/> $CHECKSTAT -s 8000000 $DIR2/$tfile || error "wrong file size""</p> <p>That's almost certainly why <a href="https://jira.whamcloud.com/browse/LU-10891" title="sanityn test 77a, 77b, 77c, 77d, 77e, 77f, 77j and 77k all fail after 32a with 'dd at *MB on client failed (2)'" class="issue-link" data-issue-key="LU-10891"><del>LU-10891</del></a> is happening after this failure.</p> <p>Two thoughts.</p> <ol> <li>Fix the test. That's easy. I'll push a patch.</li> <li>Should we remove lockless truncate? It's not safe, so it's off, so it's not used AFAIK... I'll dig in to this a little more before I go further, but I think we should consider just removing it. (Especially since its own test isn't 100% reliable. That may be a bug in lockless truncate too...)</li> </ol> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34070" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34070</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10059" title="sanityn test_32a: wrong file size" class="issue-link" data-issue-key="LU-10059"><del>LU-10059</del></a> tests: sanityn 32a restore parameters<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7dc5954230ef2933242bfcb3312f3e761021f3b2</p> <p>We introduced lockless truncate to handle cases where lots of clients were (incorrectly) opening the same file with <tt>O_TRUNC</tt>, but it caused the server to thrash. I don't know that lockless truncate has a wide applicability, but it would be good (if practical) to fix it. I think more important than lockless truncate is lockless small/unaligned write to have low latency IOPS for poorly-formed IO.</p> <p>I dug through it a bit, and the proximate cause of the failures in 32a seems to be an eviction. Can't find it in the server logs currently... If I <b>can</b> manage that (or reproduce it locally, I've got the test looping...), I'll take at least a quick look.</p> <p>I took a look at lockless truncate and convinced myself I mostly understand the basic design, but the implementation has clearly bit rotted a bit. It never worked fully in 2.x (and in fact, maybe not even in 1.8 - there's comments about it being disabled there too). I'm not sure about the lockless i/o situation more generally - That design is a bit murkier to me without investing more time.</p> <p>It's compelling stuff, but unless we want to try to do a quick-ish fix for <b>just</b> lockless truncate, I think it might be a lot of work to fix. And maybe even then.</p> <p>OK, I reproduced and have mostly figured out the lockless truncate issue.</p> <p>At least the issue I'm seeing here happens 100% of the time in this situation:</p> <p>Truncate on the same client which did the writing, and truncate a size in the middle of an extent (for example, in the test, write 8 MiB and truncate to binary 8 million. This happens if you have 8 or more OSTs).</p> <p>This results in a partial extent remaining on the client after the truncate:</p> <p>in osc_cache_truncate_start:<br/> " } else {<br/> /* this must be an overlapped extent which means only</p> <ul> <li>part of pages in this extent have been truncated.<br/> */<br/> EASSERTF(ext->oe_start <= index, ext,<br/> "trunc index = %lu/%d.\n", index, partial);<br/> /* fix index to skip this partially truncated extent */<br/> index = ext->oe_end + 1;<br/> partial = false;"</li> </ul> <p><span class="error">[...]</span></p> <p>That extent is then placed in the &oio->oi_trunc pointer, to be added back to the cache at the end of the i/o (osc_io_setattr_end) :</p> <p>osc_cache_truncate_end(env, oio->oi_trunc);</p> <p>The key thing is this:<br/> That extent still exists and is attached to the relevant LDLM lock (the one used to write it out). But since we're doing a lockless truncate, we send the punch request to the server without any LDLM locking locally, so the server tries to take the lock, and tries to call back the client write lock so it can do the truncate.</p> <p>It looks like we also have to avoid writing back this extent:<br/> " /* we need to hold this extent in OES_TRUNC state so</p> <ul> <li>that no writeback will happen. This is to avoid</li> <li>BUG 17397.</li> <li>Only partial truncate can reach here, if @size is</li> <li>not zero, the caller should provide a valid @extp. */<br/> LASSERT(*extp == NULL);<br/> *extp = osc_extent_get(ext);<br/> OSC_EXTENT_DUMP(D_CACHE, ext,<br/> "trunc at %llu\n", size);"</li> </ul> <p>This does suggest one possible route forward - This bug at least would be avoided by limiting lockless truncate to "truncate to zero".</p> <p>That's probably valuable, as you noted the O_TRUNC case is of significant interest.</p> <p>It also doesn't make sense to do "lockless" truncate for an extent that is already cached locally under a DLM lock. In that case the client should just send the truncate and advertise that it already has the lock for that extent. </p> <p>Right, but the locking has already been done by the time we learn this (cl_io_lock vs cl_io_start), and there's no guarantee the lock the client holds is sufficient for the truncate. We'd have to restart the i/o here to get the locking, and after looking at it... It's a decent bit of work.</p> <p> </p> <p>I would consider reviving lockless truncate, but only for truncate to zero. I think that would work as is, with minimal effort. I'm going to push a patch to enable lockless truncate but only for truncate to zero, along with a quick test, just to see if anything interesting happens in autotest.</p> <p>Alternately, it could both deadlock and panic in local testing.</p> <p>I'm going to table this and push a patch to add lockless truncate to ALWAYS_EXCEPT.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34081" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34081</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10059" title="sanityn test_32a: wrong file size" class="issue-link" data-issue-key="LU-10059"><del>LU-10059</del></a> tests: Disable lockless truncate test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d7f4f322514b522d2a23ce3a698e56a768e4bfbb</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34070/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34070/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10059" title="sanityn test_32a: wrong file size" class="issue-link" data-issue-key="LU-10059"><del>LU-10059</del></a> tests: sanityn 32a restore parameters<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 62b57e34d9a0df1ce4b82650d7e328db5d048b39</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/e9e439c2-eedd-11e9-add9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e9e439c2-eedd-11e9-add9-52540065bddc</a></p> <p>This hit 6x in the past week.</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/a345510a-c777-4bc1-8c30-2413be63a24a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a345510a-c777-4bc1-8c30-2413be63a24a</a></p> <p>It looks like we are still seeing this issue. We see two different errors/situations in the suite log.</p> <p>One of the errors is ‘can’t lstat’with no complaint from truncate</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 32a: lockless truncate =============================================================== 17:20:44 (1603992044) CMD: trevis-9vm6 /usr/sbin/lctl get_param -n lod.lustre-MDT0000*.stripesize CMD: trevis-9vm3.trevis.whamcloud.com params=\$(/usr/sbin/lctl get_param osc.*.lockless_truncate); [[ -z \"\" ]] && param= || param=\$(grep <<< \"\$params\"); [[ -z \$param ]] && param=\"\$params\"; while read s; do echo client \$s; done <<< \"\$param\" checking cached lockless truncate Can't lstat /mnt/lustre2/f32a.sanityn: Input/output error sanityn test_32a: @@@@@@ FAIL: wrong file size </pre> </div></div> <p>We see this error for<br/> 2.12.5.67 - <a href="https://testing.whamcloud.com/test_sets/9881eb9e-1130-4da5-9312-a4451d67c59c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9881eb9e-1130-4da5-9312-a4451d67c59c</a><br/> 2.13.55.104 - <a href="https://testing.whamcloud.com/test_sets/7ad28649-b4a5-458a-8b3f-a08820a4b85c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7ad28649-b4a5-458a-8b3f-a08820a4b85c</a></p> <p>The other error we are seeing is a truncate error and the report on different size</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 32a: lockless truncate =============================================================== 18:43:28 (1601491408) CMD: trevis-65vm4 /usr/sbin/lctl get_param -n lod.lustre-MDT0000*.stripesize CMD: trevis-65vm1.trevis.whamcloud.com params=\$(/usr/sbin/lctl get_param osc.*.lockless_truncate); [[ -z \"\" ]] && param= || param=\$(grep <<< \"\$params\"); [[ -z \$param ]] && param=\"\$params\"; while read s; do echo client \$s; done <<< \"\$param\" checking cached lockless truncate truncate: cannot truncate '/mnt/lustre/f32a.sanityn' to length 8000000: Input/output error /mnt/lustre2/f32a.sanityn has size 7340032, not 8000000 sanityn test_32a: @@@@@@ FAIL: wrong file size </pre> </div></div> <p>We see this error for<br/> 2.12.5.50 - <a href="https://testing.whamcloud.com/test_sets/7c82a5a3-67f9-4d9e-996b-e6584cbad2d3" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7c82a5a3-67f9-4d9e-996b-e6584cbad2d3</a></p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40496" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40496</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10059" title="sanityn test_32a: wrong file size" class="issue-link" data-issue-key="LU-10059"><del>LU-10059</del></a> tests: sanityn 32a error messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1ec9dafe19fe31b5a19151a33cdb388f359fa7c1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40496/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40496/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10059" title="sanityn test_32a: wrong file size" class="issue-link" data-issue-key="LU-10059"><del>LU-10059</del></a> tests: sanityn 32a error messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3274e573957e8b8a067ae28c3f7d7788d40f310e</p> <p>We see the failure occasionally. Behind the error message</p> <p>Input/output error<br/> sanityn test_32a: @@@@@@ FAIL: cached truncate - wrong file size </p> <p>is the failure of lstat("/mnt/lustre2/f32a.sanityn"). But the file's size, stat and layout look normal, observed after the test failure.</p> <p>From the debug logs on mds0 (test driver), ldlm fails to connect to oss0: (-107 is no-conn)</p> <p>00000100:02020000:11.0:1642177889.168635:0:9731:0:(client.c:1371:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff895ad1180800: operation ldlm_enqueue to node 10.6.4.23@tcp failed: rc = -107</p> <p>Slightly earlier by timestamp, the oss0 debug log has the following (op 101 is ldlm_enqueue):</p> <p>00000020:00080000:10.0:1642177889.168118:0:14028:0:(tgt_handler.c:770:tgt_request_handle()) operation 101 on unconnected OST from 12345-10.6.4.19@tcp</p> <p>Also from the dmesg on mds0:</p> <p>[ 1346.778759] LustreError: 11-0: lustre-OST0000-osc-ffff895ad1180800: operation ldlm_enqueue to node 10.6.4.23@tcp failed: rc = -107<br/> [ 1346.785968] Lustre: lustre-OST0000-osc-ffff895ad1180800: Connection to lustre-OST0000 (at 10.6.4.23@tcp) was lost; in progress operations using this service will wait for recovery to complete<br/> [ 1346.787088] LustreError: 167-0: lustre-OST0000-osc-ffff895ad1180800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.<br/> [ 1346.796872] Lustre: lustre-OST0000-osc-ffff895ad1180800: Connection restored to 10.6.4.23@tcp (at 10.6.4.23@tcp)<br/> [ 1346.951394] Lustre: DEBUG MARKER: sanityn test_32a: @@@@@@ FAIL: cached truncate - wrong file size</p> <p>It seems the mds0 and oss0 have a temporary connection error. It's unlikely due to a random network issue because other tests are OK when test_32a's fails, as we have observed many times.</p> <p>Uploaded the following files. The debug logs are denoted "xxxxx".</p> <p>sanityn.test_32a.debug_log.mds0.32a_only sanityn.test_32a.dmesg.mds0<br/> sanityn.test_32a.debug_log.oss0.32a_only sanityn.test_32a.test_log.mds0</p> <p> </p> <p>Test removed in </p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'><a href="https://jira.whamcloud.com/browse/LU-14838" title="Remove old lockless code: Truncate & contention based lockless i/o" class="issue-link" data-issue-key="LU-14838"><del>LU-14838</del></a></td> </tr> </tbody></table> </div> Related LU-10891 Development Rank 1|hzzl4n: Rank (Obsolete) 9223372036854775807 Severity