Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-16030] sanity-pcc test_45: attach more than 2 time: 0 https://jira.whamcloud.com/browse/LU-16030 Lustre <p>This issue was created by maloo for Qian Yingjin <qian@ddn.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5cddefde-7d18-45d2-9fe6-496bfef626d4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5cddefde-7d18-45d2-9fe6-496bfef626d4</a></p> <p>test_45 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>attach more than 2 time: 0 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-pcc test_45 - attach more than 2 time: 0</p> LU-16030

sanity-pcc test_45: attach more than 2 time: 0

Bug Minor Open Unresolved Qian Yingjin Maloo Wed, 20 Jul 2022 09:39:37 +0000 Mon, 1 May 2023 00:34:18 +0000 0 7 <p>It seems that this failure occurred on the newer kernel SLES 15.3 and Unubtu2004.</p> <p>+3 in past week:<br/> <a href="https://testing.whamcloud.com/sub_tests/1049c17f-fba7-4d78-9620-3e5e5c3f6b17" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/1049c17f-fba7-4d78-9620-3e5e5c3f6b17</a><br/> <a href="https://testing.whamcloud.com/sub_tests/3fe48ac1-c27c-488e-b95a-37e6abd93d9d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/3fe48ac1-c27c-488e-b95a-37e6abd93d9d</a><br/> <a href="https://testing.whamcloud.com/sub_tests/6a045690-d7a6-476c-8cf2-47d070a62de0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/6a045690-d7a6-476c-8cf2-47d070a62de0</a></p> <p>After investigate the problem, the problem is narrowed down to a generic Lustre bug even without PCC on the newer kernel (Ubuntu 2204).</p> <p>The bug can be easily reproduced by the following scripts (without PCC setup on the client):</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #!/bin/bash error() { echo <span class="code-quote">"$@"</span> exit 1 } HSMTOOL_ARCHIVE_FORMAT=v2 LCTL=$(which lctl) LFS=$(which lfs) MMAP_CAT=<span class="code-quote">"/usr/lib/lustre/tests/mmap_cat"</span> DIR=<span class="code-quote">"/mnt/lustre"</span> tdir=<span class="code-quote">"sanity-pcc.d99"</span> tfile=<span class="code-quote">"sanity-pcc.f99"</span> hsm_root=<span class="code-quote">"/mnt/pcc"</span> mntpt=<span class="code-quote">"/mnt/pcc"</span> dir=$DIR/$tdir file=$dir/$tfile id=2 cnt=30 DEBUGFS=/sys/kernel/debug/tracing CURRENT_TRACER=<span class="code-quote">"function_graph"</span> SET_FUNCTION_GRAPH=<span class="code-quote">"*:mod:lustre"</span> lctl set_param debug=cache+inode+iotrace+vfstrace+info+dlmtrace+trace+reada lctl set_param subsystem_debug=llite+ldlm+osc lctl set_param debug_mb=40000 lctl get_param debug_mb rm -rf $mntpt/* rm -rf $DIR/* rm logC rm mytrace.dat echo <span class="code-quote">"$CURRENT_TRACER"</span> > $DEBUGFS/current_tracer echo 12 > $DEBUGFS/max_graph_depth #echo <span class="code-quote">"$SET_FUNCTION_GRAPH"</span> > $DEBUG_FS/set_ftrace_filter echo '' > $DEBUG_FS/set_ftrace_filter mkdir -p $dir lctl clear dd <span class="code-keyword">if</span>=/dev/zero of=$file bs=1M count=$cnt || error <span class="code-quote">"Write $file failed"</span> lctl clear ( <span class="code-keyword">while</span> [ ! -e $DIR/sanity-pcc.99.lck ]; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=/dev/zero of=$file bs=1M count=$cnt conv=notrunc || error <span class="code-quote">"failed to write $file"</span> sleep 0.$((RANDOM % 4 + 1)) done )& wpid=$! ( <span class="code-keyword">while</span> [ ! -e $DIR/sanity-pcc.99.lck ]; <span class="code-keyword">do</span> #lctl clear echo > $DEBUGFS/trace echo 1 > $DEBUGFS/tracing_on dd <span class="code-keyword">if</span>=$file of=/dev/<span class="code-keyword">null</span> bs=1M count=$cnt & pid=$! echo <span class="code-quote">"$pid"</span> > $DEBUGFS/set_ftrace_pid wait $pid || { rc=$? cp $DEBUGFS/trace mytrace.dat echo 0 > $DEBUGFS/tracing_on echo <span class="code-quote">'nop'</span> > $DEBUGFS/current_tracer echo <span class="code-quote">"failed to read $file: rc = $rc pid = $pid"</span> lctl dk > logA error <span class="code-quote">"failed to read $file: rc = $?"</span> } sleep 0.$((RANDOM % 4 + 1)) echo 0 > $DEBUGFS/tracing_on done )& rpid=$! ( <span class="code-keyword">while</span> [ ! -e $DIR/sanity-pcc.99.lck ]; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=$file of=/dev/<span class="code-keyword">null</span> bs=1M count=$cnt iflag=direct || error <span class="code-quote">"failed to mmap_cat $file"</span> sleep 0.$((RANDOM % 4 + 1)) done <span class="code-keyword">true</span> )& rpid2=$! sleep 5 touch $DIR/sanity-pcc.99.lck wait $wpid || error <span class="code-quote">"$?: write failed"</span> wait $rpid || error <span class="code-quote">"$?: read failed: pid=$rpid"</span> wait $rpid2 || error <span class="code-quote">"$?: read2 failed"</span> </pre> </div></div> <p>The output:</p> <p> </p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 30+0 records in 30+0 records out 31457280 bytes (31 MB, 30 MiB) copied, 0.353618 s, 89.0 MB/s dd: error reading <span class="code-quote">'/mnt/lustre/sanity-pcc.d99/sanity-pcc.f99'</span>: Input/output error 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.459942 s, 9.1 MB/s 30+0 records in </pre> </div></div> <p> </p> <p> </p> <p>Three process keeping buffered read/direct I/O read from a file and write to the same file will return -EIO .</p> <p>The EIO code is returned from:<br/> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> filemap_read_page(struct file *file, struct address_space *mapping, struct page *page) { <span class="code-object">int</span> error; /* * A previous I/O error may have been due to temporary failures, * eg. multipath errors. PG_error will be set again <span class="code-keyword">if</span> readpage * fails. */ ClearPageError(page); <span class="code-comment">/* Start the actual read. The read will unlock the page. */</span> error = mapping->a_ops->readpage(file, page); <span class="code-keyword">if</span> (error) <span class="code-keyword">return</span> error; error = wait_on_page_locked_killable(page); <span class="code-keyword">if</span> (error) <span class="code-keyword">return</span> error; <span class="code-keyword">if</span> (PageUptodate(page)) <span class="code-keyword">return</span> 0; shrink_readahead_size_eio(&file->f_ra); <span class="code-keyword">return</span> -EIO; <=== <span class="code-keyword">return</span> code!! }</pre> </div></div> <p> </p> <p>Still need more time to investigate the bug....</p> <p> </p> <p>After investigate the bug, the problem should be that:</p> <p>The client has two overlapped PR DLM extent locks:</p> <ul> <li>L1 = <PR, <span class="error">[1M, 4M - 1]</span>></li> <li>L2 = <PR, <span class="error">[3M, 5M - 1]</span>></li> </ul> <p>A reader process holds L1 and read data in range <span class="error">[3M, 4M -1]</span>.</p> <p>L2 is being revoking due to the conflict access.</p> <p>Then in the ->readpage() call:</p> <p>It will unlock the page after the page is in Uptodate state.</p> <p>And lock blocking callback for L2 will clear the Uptodate state and delete the page from cache.</p> <p>Then in the old kernel, it will:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Start the actual read. The read will unlock the page. */</span> error = mapping->a_ops->readpage(filp, page); <span class="code-keyword">if</span> (unlikely(error)) { <span class="code-keyword">if</span> (error == AOP_TRUNCATED_PAGE) { put_page(page); error = 0; <span class="code-keyword">goto</span> find_page; } <span class="code-keyword">goto</span> readpage_error; } <span class="code-keyword">if</span> (!PageUptodate(page)) { error = lock_page_killable(page); <span class="code-keyword">if</span> (unlikely(error)) <span class="code-keyword">goto</span> readpage_error; <span class="code-keyword">if</span> (!PageUptodate(page)) { <span class="code-keyword">if</span> (page->mapping == NULL) { =====> /* * invalidate_mapping_pages got it */ unlock_page(page); put_page(page); <span class="code-keyword">goto</span> find_page; } unlock_page(page); shrink_readahead_size_eio(filp, ra); error = -EIO; <span class="code-keyword">goto</span> readpage_error; } unlock_page(page); } <span class="code-keyword">goto</span> page_ok; </pre> </div></div> <p>It will check whether the page was truncated and deleted from page cache via page->mapping, if so, it will retry the page read.</p> <p> </p> <p>But in the newer kernel, it deletes this check and retry.</p> <p>It seems there is no any simple solution for this bug.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> filemap_read_page(struct file *file, struct address_space *mapping, struct page *page) { <span class="code-object">int</span> error; /* * A previous I/O error may have been due to temporary failures, * eg. multipath errors. PG_error will be set again <span class="code-keyword">if</span> readpage * fails. */ ClearPageError(page); <span class="code-comment">/* Start the actual read. The read will unlock the page. */</span> error = mapping->a_ops->readpage(file, page); <span class="code-keyword">if</span> (error) <span class="code-keyword">return</span> error; error = wait_on_page_locked_killable(page); <span class="code-keyword">if</span> (error) <span class="code-keyword">return</span> error; <span class="code-keyword">if</span> (PageUptodate(page)) <span class="code-keyword">return</span> 0; shrink_readahead_size_eio(&file->f_ra); <span class="code-keyword">return</span> -EIO; } </pre> </div></div> <p> </p> <p>And I also found that it may cause short read (partial read) during the test...</p> <p>Which may be related to DDN-3569.</p> <p> </p> <p> </p> <p>According to the Linux Documents: Documentations/filkesystems/locking.rst:<br/> <br/> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ocking rules: All except set_page_dirty and freepage may block ====================== ======================== ========= =============== ops PageLocked(page) i_rwsem invalidate_lock ====================== ======================== ========= =============== writepage: yes, unlocks (see below) readpage: yes, unlocks shared writepages: set_page_dirty no readahead: yes, unlocks shared readpages: no shared write_begin: locks the page exclusive write_end: yes, unlocks exclusive bmap: invalidatepage: yes exclusive releasepage: yes freepage: yes direct_IO: isolate_page: yes migratepage: yes (both) putback_page: yes launder_page: yes is_partially_uptodate: yes error_remove_page: yes swap_activate: no swap_deactivate: no ====================== ======================== ========= =============== The filesystem must exclusively acquire invalidate_lock before invalidating page cache in truncate / hole punch path (and thus calling into ->invalidatepage) to block races between page cache invalidation and page cache filling functions (fault, read, ...).</pre> </div></div> <p> <br/> <br/> The solution should be that:<br/> we must hold exclusive invalidate_lock (mapping->invalidate_lock) (in the newer kernel) when remove pages from page cache caused by the revoking of the DLM extent lock protecting them.</p> <p>I will cook a patch to verify it later.</p> <p>Verifed that after acquired invalidate_lock during the lock blocking AST (extent lock), the test passed without -EIO.<br/> As this is a exclusive lock on the inode, it may reduce the concurrency for multiple lock blocking ASTs (from multiple stripes)...</p> <p>Yingjin, what is the path forward for this bug? Is there a fix in Lustre that can be made, possibly one already in flight for the other read cache issues, or is the only fix to submit a patch to Ubuntu to backport a fix from upstream to their kernel?</p> <blockquote> <p>fter investigate the bug, the problem should be that:<br/> The client has two overlapped PR DLM extent locks:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>L1 = <PR, [1M, 4M - 1]> L2 = <PR, [3M, 5M - 1]> </pre> </div></div> <p>A reader process holds L1 and read data in range <span class="error">[3M, 4M -1]</span>.<br/> L2 is being revoking due to the conflict access.<br/> Then in the ->readpage() call:<br/> It will unlock the page after the page is in Uptodate state.<br/> And lock blocking callback for L2 will clear the Uptodate state and delete the page from cache.</p></blockquote> <p>I recall that when a lock was being cancelled that the client would check for an overlapping read lock and attach the page to the second lock rather than invalidating it. I believe that was stopped because it added overhead for cancelling locks and the thought was that the page would be refreshed if it was accessed again. </p> <p>In addition to patching the kernel to fix this retry issue there, another option would be to do an interval tree lookup on the other DLM locks on this resource to see if there is an overlapping lock extent, and then bulk transfer all of the pages from the old lock to the new lock, rather than doing this one page at a time. This lock could also be cached in case other pages need to be evicted. </p> <p>In the common case, the client resource will only have a single lock on it, so the lookup would be fast.</p> <p>We could also examine why the client is requesting (or is granted) overlapping DLM extent locks, since that would also avoid the issue. </p> <p>"I recall that when a lock was being cancelled that the client would check for an overlapping read lock and attach the page to the second lock rather than invalidating it. I believe that was stopped because it added overhead for cancelling locks and the thought was that the page would be refreshed if it was accessed again."</p> <p>No, I believe we still do that, though I think we played with not doing so. We actually don't attach pages to locks in any way - they just live 'under' the DLM lock. We just keep page if there's a still lock covering them - check_and_discard_cb. There's no linkage from a page to a lock (and so also pages don't hold references on locks. It would be quite expensive to do so.)</p> <p>So, about the bug here, just a moment...</p> <p>So, I'm about 95% sure the bug here is fixed - on all kernels - by:<br/> <a href="https://jira.whamcloud.com/browse/LU-16649" title="EIO is possible on race with page reclaim/deletion" class="issue-link" data-issue-key="LU-16649"><del>LU-16649</del></a> llite: EIO is possible on a race with page reclaim <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50344/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50344/</a> </p> <p>The described situation is identical.</p> <p>But that fix is a bit of a hack, and the 'right' way to do it is to use the invalidate lock as Yingjin describes here. It just doesn't exist on older kernels.</p> <p>Yingjin made patches using the invalidate lock. See the series here:<br/> <a href="https://jira.whamcloud.com/browse/LU-16651" title="hold invalidate_lock when invalidating page cache under kernel 5.15+" class="issue-link" data-issue-key="LU-16651"><del>LU-16651</del></a> llite: hold invalidate_lock when invalidate cache pages / <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50371/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50423/</a></p> <p>That series ends by removing the hack from <a href="https://jira.whamcloud.com/browse/LU-16649" title="EIO is possible on race with page reclaim/deletion" class="issue-link" data-issue-key="LU-16649"><del>LU-16649</del></a> (and Panda's <a href="https://jira.whamcloud.com/browse/LU-16160" title="take ldlm lock when queue sync pages" class="issue-link" data-issue-key="LU-16160"><del>LU-16160</del></a> llite: SIGBUS is possible on a race with page reclaim / <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49647)." class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49647)</a> , but of course only on newer kernels.</p> <p>But to be clear:<br/> Yingjin's patches are an improvement, but as far as I know, we get correct operation with <a href="https://jira.whamcloud.com/browse/LU-16649" title="EIO is possible on race with page reclaim/deletion" class="issue-link" data-issue-key="LU-16649"><del>LU-16649</del></a>, including for this bug.</p> <p>Yingjin, please confirm and correct if I have anything badly wrong <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>"We could also examine why the client is requesting (or is granted) overlapping DLM extent locks, since that would also avoid the issue."</p> <p>So, just an additional note:<br/> We can't really do this. Read locks can always overlap, particularly if there's also a write lock covering part of the file. (This assumes a single stripe file for simplicity; everything I'm saying applies at the level of a single stripe, even in multi-stripe files.)</p> <p>Consider this case:</p> <p>Client 1 has a write lock from 1-3 MiB on a file<br/> Client 2 requests to read 0 - 1 MiB<br/> Server grants a read lock from 0-1 MiB on client 2 (server does not extend client 2 request due to write lock)<br/> Client 2 requests to read 1-2 MiB<br/> Server cancels write lock from client 1<br/> Server processes client 2 lock request, no conflicts<br/> Server grants read lock from 0 MiB to infinity for client 2<br/> Client 2 now has overlapping read locks.</p> <p>Also consider that server lock expansion isn't needed either, if the second read request is for 0.5 MiB to 1.5 MiB. We don't split read/write requests across dlm locks (within an OSC), for various good reasons, so this also results in a second overlapping read lock.</p> <p>To be fair, we also don't really protect against racing lock requests from different processes on the same client. We do a little bit of it - the OSC locking code can catch it - but it's racy, so sometimes we can get multiple identical read locks from parallel read requests (we can also get overlapping write lock requests from the same client when there are overlapping writes from userspace, resulting in a self-conflict, which is resolved safely with the usual lock callback mechanisms, just with the slightly odd sight of a client requesting two write locks on the same area of a file).</p> <p>I looked at fixing these overlaps and self-conflicts back when I was first working on lockahead (it's very noticeable with lockahead, particularly if you screw up in userspace and request the same lock over and over), but concluded the way it is is basically fine and not worth the trouble to fix. It happens only occasionally during parallel reads and writes and because it's rare and transient, I couldn't see a meaningful cost.</p> <p>(OK, I think this is really my final edit...)<br/> And because overlapping read locks are always possible, fixing the race that allows them with parallel reads doesn't get us much. We still have to handle that case.</p> Related LU-16651 Development Rank 1|i02uyv: Rank (Obsolete) 9223372036854775807 Severity