https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13997 Lustre <p>This issue was created by maloo for wangshilong &lt;wshilong@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/f01e487a-628b-41e2-9646-67794dd75816" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f01e487a-628b-41e2-9646-67794dd75816</a></p> <p>test_418 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>'df and lfs df inodes output mismatch: df -i: 10.9.4.8@tcp:/lustre 916924 14005 902919 2% /mnt/lustre, lfs df -i: filesystem_summary: 917922 15003 902919 2% /mnt/lustre' </pre> </div></div> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_418 - df and lfs df blocks output mismatch: df -i: 10.9.4.8@tcp:/lustre 916924 14005 902919 2% /mnt/lustre, lfs df -i: filesystem_summary: 917922 15003 902919 2% /mnt/lustre'</p> LU-13997

sanity test_418: df and lfs df blocks output mismatch

Bug Major Reopened Unresolved WC Triage Maloo always_except Fri, 25 Sep 2020 14:13:58 +0000 Thu, 2 Dec 2021 16:55:07 +0000 Lustre 2.15.0 0 10 <p> on master: <a href="https://testing.whamcloud.com/test_sessions/7efffeb4-d38b-4623-843a-bb9823e14897" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/7efffeb4-d38b-4623-843a-bb9823e14897</a></p> <p><a href="https://testing.whamcloud.com/test_sets/04bd7c7f-cce4-4f68-adfd-81cbfa3062d0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/04bd7c7f-cce4-4f68-adfd-81cbfa3062d0</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/82e1130a-47d7-4062-ac9e-cd13d44b3b7a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/82e1130a-47d7-4062-ac9e-cd13d44b3b7a</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/5d0de25e-6cbb-46a8-9c75-2bbd5c350945" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5d0de25e-6cbb-46a8-9c75-2bbd5c350945</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/253d6740-d5c2-4113-8765-731985896f6b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/253d6740-d5c2-4113-8765-731985896f6b</a></p> <p>+1 on ES6 <a href="https://testing.whamcloud.com/test_sets/5635e7ed-ef1a-4c8b-b11b-d3b4d76a91fb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5635e7ed-ef1a-4c8b-b11b-d3b4d76a91fb</a></p> <p>I noticed that check_lfs_df() function doesn't wait at all while it should do to let cached data to expire?</p> <p>I see there is a "<tt>sleep 0.2</tt>" inside the loop to wait until there is an update:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">for</span> count in {1..100}; <span class="code-keyword">do</span> cancel_lru_locks sync; sleep 0.2 </pre> </div></div> <p>While this issue has been open since 2020-09-25, it really only started failing after 2021-07-20. There was one failure on 2021-07-20, and another single failure on 2021-07-22, then between 4-10 per day since 2021-07-25. Patches landed in that date range are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>git log --oneline --before 2021-07-25 --after 2021-07-19 233344d451 LU-13417 test: generate uneven MDTs early for sanity 413 e90794af4b LU-14868 llite: revert 'simplify callback handling for async getattr' cbc62b0b82 LU-14826 mdt: getattr_name("..") under striped directory 6a4be282bb LU-14833 sec: quiet spurious gss_init_svc_upcall() message f75ff33d9f LU-14114 lnet: print device status in net show command 601c48f3ec LU-14804 nodemap: do not return error for improper ACL 188112fc80 LU-14300 quota: avoid nested lqe lookup 9f672d8a71 LU-14508 lfs: make mirror operations preserve timestamps 253a9e3bcd LU-12214 build: fix SLES build/install 9e1e29406c LU-12022 tests: error on resync failure sanity-flr 6cba1b949d LU-6142 libcfs: checkpatch cleanup of libcfs fail.c 9976d2c35d LU-6142 lnet: convert kiblnd/ksocknal_thread_start to vararg a39f078041 LU-6142 lnet: discard lnet_current_net_count a3f5aa624b LU-14217 osd-zfs: allow SEEK_HOLE/DATA only with sync 0f8db7e06a LU-14805 llite: No locked parallel DIO 1dbe63301b LU-13440 utils: fix handling of lsa_stripe_off = -1 f2a16793fa LU-14541 llite: avoid stale data reading </pre> </div></div> <p>My guess is that "<tt><a href="https://jira.whamcloud.com/browse/LU-13417" title="DNE3: mkdir() automatically create remote directory on MDS which has more space" class="issue-link" data-issue-key="LU-13417"><del>LU-13417</del></a> test: generate uneven MDTs early for sanity 413</tt>" or "<tt><a href="https://jira.whamcloud.com/browse/LU-13440" title="DNE3: limit directory default layout inheritance" class="issue-link" data-issue-key="LU-13440"><del>LU-13440</del></a> utils: fix handling of lsa_stripe_off = -1</tt>" causing some kind of difference in space usage for DNE that only shows up in "<tt>lfs df -i</tt>" (e.g. many remote entries), or possibly stray file/object deletions still happening in the background?</p> <p>I see in the logs the following lines, repeating many times during 418:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 00000004:00000024:1.0:1630450218.589991:0:617469:0:(mdt_handler.c:547:mdt_statfs()) blocks cached 10731520 granted 354615296 pending 0 free 1165684736 avail 1060012032 </pre> </div></div> <p>i.e. tgd_tot_dirty= 10731520<br/> it looks like cancel_lru_locks in check_lfs_df() doesn't flush dirty DoM pages? or the dirty pages are on another client?</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44803" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44803</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13997" title="sanity test_418: df and lfs df blocks output mismatch" class="issue-link" data-issue-key="LU-13997">LU-13997</a> tests: sanity/418 to cancel all client locks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 37505dfa7cd3a24e78c07be155481217a2258866</p> <p>This is failing about 1/25 of the test runs this week (31/695).</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44845" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44845</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13997" title="sanity test_418: df and lfs df blocks output mismatch" class="issue-link" data-issue-key="LU-13997">LU-13997</a> tests: sanity/418 reproducer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2aa6902a3dd18e9f4b97fafa4e4f82c6bd6c258d</p> <p>I'm not sure if this is relevant or not, but it appears that this problem hits much more frequently on ldiskfs than ZFS. For ZFS there were 3/220 failures (1/73) in the past 4 weeks, and no failures in the past 2 weeks. For ldiskfs, there were 84 / 1403 failures (1/16) with 35 in the past week (including the 95 test passes for the two patches for this issue).</p> <p>Alex wrote:</p> <blockquote> <p>I noticed the following message on MDT, repeating many times:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000024:1.0:1630450218.589991:0:617469:0:(mdt_handler.c:547:mdt_statfs()) blocks cached 10731520 granted 354615296 pending 0 free 1165684736 avail 1060012032 </pre> </div></div> <p>where cached are blocks reported "dirty" by the client my understanding is that it's DoM data to be flushed</p></blockquote> <p>Interesting. Looking at <tt>mdt_statfs()</tt>, it subtracts (<tt>tgd-&gt;tgd_tot_dirty + tgd-&gt;tgd_tot_pending</tt>) from <tt>os_bavail</tt> before returning it to the client, but when the clients do "<tt>lfs df</tt>" they ignore the MDT <tt>os_bavail</tt> value when calculating available space, and only OST <tt>os_bavail</tt> is used (for better or worse), and that is the inconsistent value...</p> <p>This really points to some other bug, that MDC/MDT grant is not calculated correctly, or that dirty DoM files are not being flushed from the clients properly, or something, since test_418 is waiting at least 20s for the output to stabilize, plus another 50s or more for test_414..test_417.</p> <p>It looks like <tt>test_413a-&gt;generate_uneven_mdts()</tt> is creating a lot of DoM files, and test_413z is deleting them, since v2_14_52-97-g233344d451 on master (2021-07-25). This aligns with the start of high-frequency test_418 failures on master. In fact, the first "recent" failure was 2021-07-23 on patch <a href="https://review.whamcloud.com/44384" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44384</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13417" title="DNE3: mkdir() automatically create remote directory on MDS which has more space" class="issue-link" data-issue-key="LU-13417"><del>LU-13417</del></a> test: generate uneven MDTs early for sanity 413</tt>" <b>before it landed</b>, which is the patch that introduced <tt>generate_uneven_mdts()</tt>. Before that landed test_418 only failed 0/1 times per month.</p> <p>So it looks like patch 44803 may avoid this test failure, but there is something strange happening with dirty DoM files...</p> <p>check_lfs_df() function cancels only local locks, but iirc sanity may use two clients?</p> <blockquote> <p>check_lfs_df() function cancels only local locks, but iirc sanity may use two clients?</p></blockquote> <p>Sure, but the dirty DoM data should be flushed by itself far sooner than 60-70s since the files were written. I suppose it might be possible that the data is actually written, but the client did not generate any more RPCs, so the <tt>tgt_tot_dirty</tt> was not updated on the server since the last write? If that is the case, then the fix in patch 44384 is the right one and the problem is only in the test.</p> <p>I just want to make sure we aren't missing some other problem, where the DoM file writes are never being sent to the MDT unless the lock is cancelled. Similar bugs have happened in the past, and writes from minutes/hours earlier are lost if the client is evicted or fails. It would be possible to create a test that writes some DoM files, waits 60s (2x the <tt>vm.dirty_expire_centisecs=3000</tt> interval for data, and 12x the <tt>vm.dirty_writeback_centisecs=500</tt> interval for metadata), evicts the client, and then checks that the files contain valid data.</p> <p>here is a simple script...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + <span class="code-keyword">for</span> ((i=0; i &lt; 100; i++)); <span class="code-keyword">do</span> + $LFS setstripe -E 1M -L mdt $DIR/$tfile-$i || error <span class="code-quote">"can't setstripe"</span> + dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/$tfile-$i bs=128k count=1 || error <span class="code-quote">"can't dd"</span> + done + start=$SECONDS + <span class="code-keyword">for</span> ((i=0; i &lt; 100; i++)); <span class="code-keyword">do</span> + grant_dirty=$($LCTL get_param -n mdt.*.tot_dirty | + awk <span class="code-quote">'{sum=sum+$1} END {print sum}'</span>) + vm_dirty=$(awk <span class="code-quote">'/Dirty:/{print $2}'</span> /proc/meminfo) + echo <span class="code-quote">"dirty after $((SECONDS-start)): $grant_dirty in grants, $vm_dirty in system"</span> + sleep 2 + (( vm_dirty == 0 )) &amp;&amp; <span class="code-keyword">break</span> + done </pre> </div></div> <p>... and its output:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> dirty after 0: 1966080 in grants, 12292 in system dirty after 2: 1966080 in grants, 12392 in system dirty after 4: 1966080 in grants, 12392 in system dirty after 6: 1966080 in grants, 12424 in system dirty after 8: 1966080 in grants, 12176 in system dirty after 10: 1966080 in grants, 12176 in system dirty after 12: 1966080 in grants, 12176 in system dirty after 14: 1966080 in grants, 12176 in system Lustre: lustre-OST0000-osc-ffff8801ee82e000: disconnect after 22s idle dirty after 16: 1966080 in grants, 11712 in system dirty after 18: 1966080 in grants, 11648 in system dirty after 20: 1966080 in grants, 11648 in system dirty after 22: 1966080 in grants, 11648 in system dirty after 24: 1966080 in grants, 11664 in system dirty after 26: 1966080 in grants, 11664 in system dirty after 28: 1966080 in grants, 11664 in system dirty after 30: 3579904 in grants, 2064 in system dirty after 32: 3579904 in grants, 2064 in system dirty after 34: 155648 in grants, 440 in system dirty after 37: 155648 in grants, 416 in system dirty after 39: 155648 in grants, 416 in system dirty after 41: 155648 in grants, 416 in system dirty after 43: 155648 in grants, 504 in system dirty after 45: 155648 in grants, 16 in system dirty after 47: 155648 in grants, 16 in system dirty after 49: 155648 in grants, 16 in system dirty after 51: 155648 in grants, 16 in system dirty after 53: 155648 in grants, 0 in system </pre> </div></div> <p>i.e. all dirty pages had been flushed, but MDT still counts few pages dirty</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>: dirty after 53: 155648 in grants, 0 in system </pre> </div></div> <p>all dirty pages had been flushed, but MDT still counts few pages dirty</p></blockquote> <p>Thanks for testing this. It looks like the test with your patch should fix this problem.</p> <p>actually now I doubt the patch will really help as nothing I tried (cancel_lru_locks, extra RPC, waiting some time) change tot_dirty on MDT side.</p> <p>the problem is that grants info is passed with OST_READ/OST_WRITE. so even with no actual dirty pages on the client side MDT doesn't get any updates and keeps tot_total_dirty &gt; 0 which in turn affects statfs calculation.</p> <p>another observation is that grant shrink seem to happen (and handled) to OSC-OST only.</p> <p>We need a separate patch (separate ticket?) for grant shrink to the MDTs </p> <p>but grant shrinking mechanism has own limits too:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* Don't shrink <span class="code-keyword">if</span> we are already above or below the desired limit * We don't want to shrink below a single RPC, as that will negatively * impact block allocation and <span class="code-object">long</span>-term performance. */ <span class="code-keyword">if</span> (target_bytes &lt; cli-&gt;cl_max_pages_per_rpc &lt;&lt; PAGE_SHIFT) target_bytes = cli-&gt;cl_max_pages_per_rpc &lt;&lt; PAGE_SHIFT; <span class="code-keyword">if</span> (target_bytes &gt;= cli-&gt;cl_avail_grant) { spin_unlock(&amp;cli-&gt;cl_loi_list_lock); RETURN(0); </pre> </div></div> <p>add an expection for MDT?</p> <blockquote> <p>add an expection for MDT?</p></blockquote> <p>No, I'd rather maintain good performance than make "df" totally consistent across all the clients/MDTs. Having <em>some</em> grant shrinking on the MDTs would still be desirable, but is outside the scope of this ticket. I filed <a href="https://jira.whamcloud.com/browse/LU-15010" title="grant shrink for MDTs" class="issue-link" data-issue-key="LU-15010"><del>LU-15010</del></a> for this.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44803/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44803/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13997" title="sanity test_418: df and lfs df blocks output mismatch" class="issue-link" data-issue-key="LU-13997">LU-13997</a> tests: sanity/418 to cancel all client locks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e8ee5aa0339b06aa53dc00f4911e2019a3ef3a0</p> <p>It looks like this is still failing on master, for example:<br/> <a href="https://testing.whamcloud.com/test_sets/2121a4d8-1472-473c-875c-4b3b1288b726" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2121a4d8-1472-473c-875c-4b3b1288b726</a></p> <p>+1 on master - <a href="https://testing.whamcloud.com/test_sets/1c8664ee-840b-453d-bbcd-c80e7b75d7d4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1c8664ee-840b-453d-bbcd-c80e7b75d7d4</a></p> <p>I hope <a href="https://review.whamcloud.com/#/c/44956/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/44956/</a> can help</p> <p>+1 on master - <a href="https://testing.whamcloud.com/test_sets/302baca3-ea35-4824-a7fe-d4a6995111d7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/302baca3-ea35-4824-a7fe-d4a6995111d7</a></p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45231" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45231</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13997" title="sanity test_418: df and lfs df blocks output mismatch" class="issue-link" data-issue-key="LU-13997">LU-13997</a> tests: fix sanity test_418 lock cancellation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4bb910bba91199af21199080c5f3eb1070c59c9</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45231/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45231/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13997" title="sanity test_418: df and lfs df blocks output mismatch" class="issue-link" data-issue-key="LU-13997">LU-13997</a> tests: fix sanity test_418 lock cancellation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c0596c4d8af3bce6565c0e16a1469fcc65191404</p> <p>Landed for 2.15</p> <p>The landed patch only avoids the test failure, it does not actually fix it, which is why it is marked <tt>always_except</tt> for now. </p> Related LU-15105 LU-13125 LU-15010 Development Rank 1|i01atb: Rank (Obsolete) 9223372036854775807 Severity