https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15431 Lustre <p>FLR doesn't hold layout lock (LL) during IO (OST_READ) operations, but verify layout at the beginning of IO and at the end of IO.<br/> the problem is that pages get Uptodate flag from brw_interpret context while the layout is checked for changes later. the pages are still marked uptodate till that point.<br/> three processes:</p> <ul> <li>1st modifying the file</li> <li>2nd initiated OST_READ and going to repeat OST_READ due to layout change</li> <li>3rd which finds result of OST_READ (stale data) in ll_do_fast_read()</li> </ul> LU-15431

read may return stale data from replicated file

Bug Critical Resolved Fixed Alex Zhuravlev Alex Zhuravlev flr-improvement Tue, 11 Jan 2022 09:39:04 +0000 Mon, 27 Nov 2023 11:15:35 +0000 Fri, 14 Jul 2023 12:46:10 +0000 Lustre 2.16.0 0 10 <p>Gentelmens, please have a look</p> <p>A second, more serious case can happen:</p> <ul class="alternate" type="square"> <li>1st client writes to file, MDS marks some replica(s) stale</li> <li>2nd client has cached OST extent lock of (now) stale objects</li> <li>2nd client <b>only</b> fast-reads from pages cached under extent lock of stale object (eg. 4KB/2KB/1KB sized/aligned reads).</li> </ul> <p>Because fast reads do not check DLM locks or layout version (they assume that if a page is in cache that it is valid), then there is nothing on the second client that will trigger layout refresh or reading from the primary OST object. </p> <p>At some point, the 2md client would run out of cached pages, or possibly have a read that crosses a page boundary that is not "fast" and would check the layout version so that reads go to the primary object, but this could happen seconds after the cached object became stale.</p> <p>It also isn't clear if there is a mechanism for the layout update on the client to cause all cached pages on the stale replica objects to be flushed from cache, or if only OST reads for uncached pages access the primary replica objects?</p> <p>Yuck, that’s true.</p> <p>We definitely need to add a layout version check to the fast read code, but we also need to add some way to flush the old layout and data.  That I am not sure how best to do - layout changes very carefully don’t flush everything today, but perhaps they should.</p> <p>It would be something like “I detect a layout version change, so I flush all local data”.</p> <p>I’m also a bit concerned about pending writes in cache - the FLR state cycle for the layout (read only, pending, writable) covers ongoing writes nicely, but doesn’t seem to handle dirty data already in cache.  It seems like maybe losing the layout lock should cause a flush to disk of any dirty data for the file?</p> <p>And maybe losing the layout lock should just cause a complete cache flush/destroy.  Yuck.</p> <p>So today, losing the layout lock ensures any ongoing I/O is restarted.  I think that’s handled ok.  But it does nothing for cached data, dirty or clean.</p> <p>The layout lock is already required for doing I/O to a file.  Perhaps losing it should just cause a complete cache flush for that file?  Honestly this seems like the only safe option.  Adding version checks doesn’t solve the “dirty data in cache” problem and it could leave data and locks linked to the old layout staying in cache in a weird state where they were inaccessible but still present.</p> <p>I’m not quite sure how that flush would interact with ongoing I/Os, but I don’t think I see another safe way to do this.</p> <p>And to answer the unasked question about the cost of doing that - yes, it would be expensive and slow if a file is open many places.  But if a file is open many places why are you changing the layout?  Ugh - I note this interacts poorly with SEL, which would change the layout of a file open many places and rely on not destroying the cache to keep it reasonable.</p> <p>I don’t know if we can respect that (and I don’t know if SEL has ever come in to real use.  I suspect not.).</p> <p>probably cache invalidation should be driven indirectly by layout version change on OST, not by layout lock itself.</p> <blockquote><p>I’m also a bit concerned about pending writes in cache</p></blockquote> <p><a href="https://jira.whamcloud.com/browse/LU-15300" title="mirror resync can cause EIO to unrelated applications" class="issue-link" data-issue-key="LU-15300"><del>LU-15300</del></a></p> <p>Hmm, I ran a manual test to see if I could create the "fast read from stale mirror" problem, but it seems to be working as it should. Steps were roughly as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client1# cp /etc/hosts /mnt/testfs/flr client1# lfs mirror extend -N -c1 /mnt/testfs/flr client1# while sleep 1; do dd if=/mnt/testfs/flr bs=20 count=1; done &amp; [verify via ldlm.namespaces.*.lru_size lock is on mirror#1] client2# lfs mirror write -N2 -i /etc/passwd /mnt/testfs/flr [note "lfs mirror write" does not mark mirror#1 stale] [client1 still reading /etc/hosts from mirror#1] client2# lfs setstripe --comp-set -I 65537 --comp-flags stale /mnt/testfs/flr [client1 immediately changed to /etc/passwd data] </pre> </div></div> <p>So it appears this is working correctly, and client1 either had its DLM lock on the stale object cancelled, or its layout lock update caused it to dump the local page cache for the stale object. IMHO, having the OST extent locks on the stale objects cancelled would be more robust, as this avoids dependency on the layout lock being updated on the client and any interaction with fast reads. </p> <p>Side notes:</p> <ul class="alternate" type="square"> <li>"<tt>--comp-set</tt>" is not listed in the "<tt>lfs setstripe</tt>" usage message, so does not appear in tab completion (fixed in patch <a href="https://review.whamcloud.com/41423" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41423</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14385" title="add more test cases to verify &quot;lfs setstripe&quot; options" class="issue-link" data-issue-key="LU-14385"><del>LU-14385</del></a> tests: verify lfs setstripe comp-flags and flags options</tt>" on master).</li> <li>it would be useful to have an "<tt>lfs mirror</tt>" equivalent to mark a mirror stale</li> <li>it would be useful to have an option to "<tt>lfs mirror write</tt>" to mark the other mirrors stale. Unlike normal writes, this would allow selecting a specific mirror for write.</li> <li>this test case is the only one so far that I wanted to be able to split prefer-read from prefer-write, to verify that the read was mirror#1 and the write was mirror#2</li> <li>there wasn't an easy way to verify which mirror the client1 reads were using and which objects pages were cached. I cleared the DLM cache and used <tt>lru_count</tt> to see which OSC had a lock, but not great. (<a href="https://jira.whamcloud.com/browse/LU-14858" title="kernfs tree to dump/traverse ldlm locks" class="issue-link" data-issue-key="LU-14858">LU-14858</a>)</li> <li>the <tt>llite.*.dump_page_cache</tt> output only showed the MDT FID and not the OST FID used.</li> <li>it would be very useful if dump_page cache had a "comment" line at the top to describe the fields</li> </ul> <p>I don't think we need to cancel locks for every version update, since that happens frequently. Only for updates that mark components stale, which is rarely. </p> <p>So what I've learned so far is specifically the setstripe command to change the flags</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setstripe --comp-set -I 65537 --comp-flags stale /mnt/testfs/flr </pre> </div></div> <p>Appears to trigger a flush.</p> <p>Specifically, it calls <tt>llapi_file_flush()</tt> (internally <tt>ioctl(LL_IOC_DATA_VERSION)</tt>), which sets <tt>LL_DV_WR_FLUSH</tt> in a command it sends to all the OSTs.</p> <p>And that's turned in to:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">                 <span class="code-keyword">if</span> (dv-&gt;dv_flags &amp; LL_DV_WR_FLUSH)                         oa-&gt;o_flags |= OBD_FL_FLUSH; </pre> </div></div> <p>Which causes the OST to flush.</p> <p>So changing the prefer flag manually doesn't work.</p> <p>Andreas, your suggestion to expose the mirror write and mirror read 'prefer' flags separately suddenly seems relevant, because I need to read data from one mirror then write data to the other without changing the layout.  I could also achieve this by taking one mirror temporarily offline but that's kind of challenging on my setup...  I'll think about it.</p> <p>I see that <tt>llapi_file_flush()</tt> is also called in <tt>mirror_extend_file()</tt>, but <b>not</b> in <tt>llapi_mirror_resync_many()</tt>, which would be a simple change to make, but doesn't solve the whole problem. Alternately, the resync should probably be bracketed with an <tt>llapi_get_data_version()</tt> call at the start/end so that it can verify that the data hasn't changed during resync? Otherwise, I don't see how we are certain that the file hasn't been modified since the resync started?</p> <p><b>aside</b>: the use of <tt>llapi_get_data_version()</tt> in <tt>lsom_update_one()</tt> for the "flush client data" side effect should be replaced with <tt>llapi_file_flush()</tt> to make this explicit, and avoid problems in the future.</p> <p>I <b>believe</b> mirror resync is going to depend on the FLR state over on the MDS, that state machine with the cycle from readonly to stale, etc.  So it doesn't need to verify anything - It controls the state until the resync is complete, then once it's complete, well, things can change after that.</p> <p>That's just what I <b>believe</b>, though.</p> <p>OK, I (finally) found it.  We're fine for reads, though I still need to double check on writes.  The comment in ll_readpage needs to be corrected.</p> <p>I noticed when a mirror became stale, the data stayed in cache.  But the stale data was being reliably destroyed at the start of the next I/O, which confused me.  I finally figured it out.</p> <p>When the layout lock has been lost, when we next <b>open</b> the file (ie, before we go to read it), we end up calling in to ll_iget - it looks like we clear some part of the object metadata, I'm not quite sure what - and that calls in to the layout conf code.</p> <p>That eventually calls in to lov_layout_change, which calls cl_object_prune and destroys all of the OSC objects associated with the file, which destroys the cached pages.  All before the 'read' code starts.  This <b>should</b> also work for a file which is held open because I believe Lustre should force a dentry refresh, etc, which should have the same effect?</p> <p>I'll talk about writes in a second.</p> <p>As I reflect on pending writes, I realize I forgot the basic FLR lifecycle.</p> <p>If there is a pending write (ie, dirty data), the mirrors can't be in sync.  The I/O which created that dirty data brings the mirrors out of sync.  And bringing them back in to sync <b>does</b> flush all the data - that's a requirement when implementing lfs mirror resync.</p> <p>So writes are fine too.  I think this is all fine except that comment is misleading (and probably a leftover from an earlier design).</p> <p>please check the original description - it was about early setting of PG_uptodate before the 2nd layout check. if the 2nd check is not required, then why is it here?</p> <blockquote> <p>When the layout lock has been lost, when we next open the file (ie, before we go to read it), we end up calling in to ll_iget - it looks like we clear some part of the object metadata, I'm not quite sure what - and that calls in to the layout conf code.</p></blockquote> <p>So this depends on the client being properly notified about the layout lock and canceling its own cache. It would be more robust if the OST extent lock was revoked from objects in the stale mirror(s) at the time they are marked stale, so that the client couldn't continue to read from the old objects.</p> <p>This won't help in the case of a totally isolated client (i.e. not getting callbacks from either MDS or OSS), but it does help in the case of the client being evicted/incommunicado with the MDS, but still able to contact the OSS to read data.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46273" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46273</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15431" title="read may return stale data from replicated file" class="issue-link" data-issue-key="LU-15431"><del>LU-15431</del></a> tests: a demo<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a6662c7ba1d1785e10bfacbc564a0d9d3ed4f44c</p> <p>please, have a look at <a href="https://review.whamcloud.com/#/c/46273/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/46273/</a><br/> basically the test reads inbetween llapi_file_flush() and layout merge in lod_xattr_set(), then repeat the same read in the same process with the same old fd and gets old data. unless I missed something.<br/> the output:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 48.723121] Lustre: Mounted lustre-client AAAA written: osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 fail_loc=0x16f fail_val=4 create replicate in background [ 49.049398] LustreError: 5192:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 16f sleeping <span class="code-keyword">for</span> 4ms [ 49.150019] LustreError: 5192:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 16f awake [ 49.154344] lustre-OST0000: OST_READ from [0x100000000:0x3:0x0] read right before layout merge, after data flush [ 49.549402] lustre-OST0001: OST_READ from [0x100010000:0x2:0x0] wait <span class="code-keyword">for</span> replication replication done BBBB written fast read from pagecache DATA: AAAA DATA: AAAA read via <span class="code-keyword">new</span> open(2) [ 51.576003] lustre-OST0000: OST_READ from [0x100000000:0x3:0x0] DATA: BBBB [ 51.586037] lustre-OST0000: OST_READ from [0x100000000:0x3:0x0] DATA: BBBB </pre> </div></div> <p>here is another test, with no injections with cfs_fail:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> test_2000() { local tf=$DIR/$tfile local osts=$(comma_list $(osts_nodes)) mkdir -p $MOUNT2 &amp;&amp; mount_client $MOUNT2 # to make replica on ost1 preferred <span class="code-keyword">for</span> <span class="code-keyword">new</span> writes do_nodes $osts \ $LCTL set_param osd*.*OST*.nonrotational=0 do_nodes $osts \ $LCTL set_param osd*.*OST0001*.nonrotational=1 $LFS setstripe -c1 -i0 $tf || errro <span class="code-quote">"can't create $tf"</span> echo <span class="code-quote">"AAAA"</span> &gt;$tf $LFS mirror extend -N -o1 $tf || error <span class="code-quote">"can't make replica"</span> log2 <span class="code-quote">"replicated file created"</span> cancel_lru_locks mdc cancel_lru_locks osc log2 <span class="code-quote">"open(O_RDONLY) and first read from OST"</span> $MULTIOP $tf oO_RDONLY:P4_z0P4_z0P4c &amp; PID=$! sleep 1 log2 <span class="code-quote">"first read complete"</span> echo <span class="code-quote">"BBBB"</span> &gt;/tmp<span class="code-comment">//$tfile </span> dd <span class="code-keyword">if</span>=/tmp/$tfile of=$DIR2/$tfile conv=notrunc &gt;&amp;/dev/<span class="code-keyword">null</span> || error <span class="code-quote">"can't write BBBB"</span> log2 <span class="code-quote">"BBBB written which made replica on ost1 stale"</span> log2 <span class="code-quote">"fast read from pagecache in the original process"</span> kill -USR1 $PID sleep 1 log2 <span class="code-quote">"read via $DIR2 <span class="code-keyword">new</span> open(2)"</span> $MULTIOP $DIR2/$tfile oO_RDONLY:P4c log2 <span class="code-quote">"fast read from pagecache after 5s in the original process"</span> sleep 5 kill -USR1 $PID wait $PID log2 <span class="code-quote">"read via $DIR <span class="code-keyword">new</span> open(2)"</span> $MULTIOP $tf oO_RDONLY:P4c } run_test 2000 <span class="code-quote">"test2"</span> </pre> </div></div> <p>and the output:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == sanity-flr test 2000: test2 =========================== 20:26:32 (1642969592) [ 47.814975] Lustre: DEBUG MARKER: == sanity-flr test 2000: test2 =========================== 20:26:32 (1642969592) Starting client: tmp.y4sbWzerEx: -o user_xattr,flock tmp.y4sbWzerEx@tcp:/lustre /mnt/lustre2 [ 47.982943] Lustre: Mounted lustre-client osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 [ 48.133925] Lustre: DEBUG MARKER: replicated file created [ 48.175340] Lustre: DEBUG MARKER: open(O_RDONLY) and first read from OST DATA: AAAA in 6318 [ 49.196269] Lustre: DEBUG MARKER: first read complete [ 49.222184] Lustre: DEBUG MARKER: BBBB written which made replica on ost1 stale [ 49.229754] Lustre: DEBUG MARKER: fast read from pagecache in the original process DATA: AAAA in 6318 [ 50.245148] Lustre: DEBUG MARKER: read via /mnt/lustre2 <span class="code-keyword">new</span> open(2) DATA: BBBB in 6326 [ 50.261963] Lustre: DEBUG MARKER: fast read from pagecache after 5s in the original process DATA: AAAA in 6318 [ 55.279692] Lustre: DEBUG MARKER: read via <span class="code-keyword">new</span> open(2) DATA: BBBB in 6330 </pre> </div></div> <p>notice read via 2nd client (/mnt/lustre2) gets BBBB and still after that the first client reads AAAA again (I checked the logs - it was fast read path).</p> <p>I used a trivial change to multiop: command P reads &lt;num&gt; bytes and prints as a string.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46282" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46282</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15431" title="read may return stale data from replicated file" class="issue-link" data-issue-key="LU-15431"><del>LU-15431</del></a> llite: skip fast reads if layout is invalid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 371fa1b9153a0ea765f5aea73b35e00a25be0b97</p> <blockquote> <p>So this depends on the client being properly notified about the layout lock and canceling its own cache. It would be more robust if the OST extent lock was revoked from objects in the stale mirror(s) at the time they are marked stale, so that the client couldn't continue to read from the old objects.</p></blockquote> <p>I think there are two options:</p> <ul> <li>update LV (layout version) on now-stale OST objects, which in turn must cancel all OST locks</li> <li>just stop to trust pagecache upon LL cancellation (implmented in the patch above)</li> </ul> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a> </p> <p>What are the next steps here? I thought we went with option 2, however with your patch above, shadow is saying -1. Can you please advise?</p> <blockquote><p>What are the next steps here? I thought we went with option 2, however with your patch above, shadow is saying -1. Can you please advise?</p></blockquote> <p>it'd be great it Patrick could step in and review the patch.</p> <p>I'm not 100% sure we've closed all the races here, but I think the patch is good.  Just noting we might still have something that could happen, not 100% sure.</p> <p>I'm 100% sure hte patch closes at least one hole <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/46282/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/46282/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15431" title="read may return stale data from replicated file" class="issue-link" data-issue-key="LU-15431"><del>LU-15431</del></a> llite: skip fast reads if layout is invalid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fe2fafa1af7edc251009e3fbd46665e05573bf4a</p> <p>Landed for 2.16</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53247" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53247</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15431" title="read may return stale data from replicated file" class="issue-link" data-issue-key="LU-15431"><del>LU-15431</del></a> llite: skip fast reads if layout is invalid<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 2cfbd62a4bbcdb59c8a54ddc49b8617bbb363fc4</p> Duplicate LU-15434 Related LU-11695 Development Epic Link EX-4394 Rank 1|i02ebb: Rank (Obsolete) 9223372036854775807 Severity