https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13017 Lustre <p>sanity test_56ra fails with ' '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST'. We’ve seen this error twice for 2.12.3.31; <a href="https://testing.whamcloud.com/test_sets/93cddba8-0d76-11ea-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/93cddba8-0d76-11ea-98f1-52540065bddc</a> and <a href="https://testing.whamcloud.com/test_sets/3d7c7eea-0d93-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3d7c7eea-0d93-11ea-8e77-52540065bddc</a>. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 56ra: check lfs find -size -lazy works for data on OSTs =============================== 10:36:52 (1574332612) Before: 6056 After: 6069 3 osc.lustre-OST0000-osc-ffff8c52a0a6c000.stats= snapshot_time 1574332613.130677340 secs.nsecs req_waittime 3629 samples [usec] 405 4908409 125708950 217095123014804 req_active 3630 samples [reqs] 1 16 10917 64837 ldlm_glimpse_enqueue 453 samples [reqs] 1 1 453 453 ldlm_extent_enqueue 518 samples [reqs] 1 1 518 518 read_bytes 10 samples [bytes] 0 1048576 4198400 4380984082432 write_bytes 729 samples [bytes] 4 1048576 465385120 468655951772936 ost_setattr 30 samples [usec] 493 2458360 5470309 7460871612619 ost_read 10 samples [usec] 770 112506 334235 27947789183 ost_write 729 samples [usec] 814 4301470 93802565 175458110296977 ost_connect 29 samples [usec] 655 12195 57083 265453679 ost_disconnect 17 samples [usec] 1029 3311 31912 67557322 ost_punch 511 samples [usec] 503 954851 1266293 911962877985 ost_statfs 24 samples [usec] 405 1646 17727 14510469 ost_sync 4 samples [usec] 471514 1920497 3598304 4653850793070 ost_quotactl 16 samples [usec] 414 838 8192 4340976 ldlm_cancel 952 samples [usec] 533 39037 11498758 185050546552 obd_ping 336 samples [usec] 760 4908409 8456427 28320522807435 osc.lustre-OST0001-osc-ffff8c52a0a6c000.stats= … obd_ping 364 samples [usec] 735 1157343 2386059 1579049113871 osc.lustre-OST0006-osc-ffff8c52a0a6c000.stats= snapshot_time 1574332613.130931617 secs.nsecs req_waittime 2363 samples [usec] 368 10241895 31650509 113679647081289 req_active 2364 samples [reqs] 1 15 6902 40110 ldlm_glimpse_enqueue 928 samples [reqs] 1 1 928 928 ldlm_extent_enqueue 14 samples [reqs] 1 1 14 14 read_bytes 2 samples [bytes] 0 0 0 0 write_bytes 19 samples [bytes] 1 1048576 12837273 13249571414835 ost_setattr 31 samples [usec] 459 10241895 14509301 109946640240861 ost_read 2 samples [usec] 1823 2294 4117 8585765 ost_write 19 samples [usec] 814 543145 1723935 429765970407 ost_connect 27 samples [usec] 450 12275 51577 247601705 ost_disconnect 14 samples [usec] 999 41957 63004 1800765226 ost_punch 9 samples [usec] 622 73011 79728 5336508958 ost_statfs 24 samples [usec] 368 2161 14597 12673415 ost_quotactl 16 samples [usec] 388 462 6778 2878618 ldlm_cancel 927 samples [usec] 411 38371 11153645 173370727119 obd_ping 352 samples [usec] 759 1625436 2970385 3116552824601 sanity test_56ra: @@@@@@ FAIL: '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST </pre> </div></div> <p>We’ve seen an error message that may be related for Ubuntu clients ' '/usr/bin/lfs find -size 5 -type f /mnt/lustre/d56ra.sanity' should send 14 glimpse RPCs to OST'; <a href="https://testing.whamcloud.com/test_sets/138a655c-0d5c-11ea-9487-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/138a655c-0d5c-11ea-9487-52540065bddc</a> for patch <a href="https://review.whamcloud.com/36405/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36405/</a>. From the suite_log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 56ra: check lfs find -size -lazy works for data on OSTs =============================== 17:50:11 (1574445011) Before: 4861 After: 4873 3 osc.lustre-OST0000-osc-ffff9a29368bb000.stats= snapshot_time 1574445012.373146631 secs.nsecs req_waittime 2983 samples [usec] 355 2937458 64237903 59715964412813 req_active 2983 samples [reqs] 1 13 11657 81779 ldlm_glimpse_enqueue 277 samples [reqs] 1 1 277 277 ldlm_extent_enqueue 515 samples [reqs] 1 1 515 515 read_bytes 2 samples [bytes] 4096 4096 8192 33554432 write_bytes 569 samples [bytes] 3 4194304 498256074 1247211946247316 ost_setattr 33 samples [usec] 363 608888 739594 375188097832 ost_read 2 samples [usec] 455 466 921 424181 ost_write 569 samples [usec] 554 2937458 49108763 59029078413847 ost_connect 13 samples [usec] 464 6183 24328 96647710 ost_disconnect 9 samples [usec] 755 1623 11362 14999198 ost_punch 511 samples [usec] 355 43985 346442 4026093074 ost_statfs 12 samples [usec] 357 1074 5859 3259503 ldlm_cancel 774 samples [usec] 390 63347 12515513 274346845595 obd_ping 267 samples [usec] 502 117510 544749 18808100599 osc.lustre-OST0001-osc-ffff9a29368bb000.stats= … obd_ping 265 samples [usec] 472 4485131 4896837 20121333347081 mdc.lustre-MDT0000-mdc-ffff9a29368bb000.stats= snapshot_time 1574445012.413685221 secs.nsecs req_waittime 938787 samples [usec] 305 1810374 538301391 13512208660843 req_active 938787 samples [reqs] 1 9 961715 1042401 ldlm_glimpse_enqueue 120 samples [reqs] 1 1 120 120 ldlm_ibits_enqueue 429407 samples [reqs] 1 1 429407 429407 mds_getattr 124 samples [usec] 337 1111 47863 19161519 mds_getattr_lock 5898 samples [usec] 470 7696 3084370 1885670342 mds_close 116757 samples [usec] 305 892872 46303939 1088509049017 mds_readpage 352 samples [usec] 734 33119 1049972 8758729676 mds_statfs 14 samples [usec] 355 866 6123 2898905 mds_sync 6 samples [usec] 1153 40361 46512 1636587052 mds_getxattr 28 samples [usec] 339 2024 26071 30757607 mds_hsm_state_set 8 samples [usec] 1150 1358 10021 12585899 ldlm_cancel 4370 samples [usec] 331 105790 2428541 42683139111 obd_ping 204 samples [usec] 446 3724 188084 212073474 seq_query 1 samples [usec] 472 472 472 222784 Before: 4873 After: 4885 3 Before: 4885 After: 4900 3 sanity test_56ra: @@@@@@ FAIL: '/usr/bin/lfs find -size 5 -type f /mnt/lustre/d56ra.sanity' should send 14 glimpse RPCs to OST </pre> </div></div> <p>We’ve also seen this test fail in interop testing.</p> <p>There is nothing interesting in the node console logs. </p> LU-13017

sanity test 56ra fails with ' '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST'

Bug Major Resolved Fixed Andreas Dilger James Nunez Tue, 26 Nov 2019 16:08:35 +0000 Sat, 4 Mar 2023 21:58:47 +0000 Thu, 16 Apr 2020 05:26:26 +0000 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.14.0 0 7 <p>Gu Zheng</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Sure, Peter</p> <p>I don't think the patch <a href="https://review.whamcloud.com/36405/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36405/</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11575" title="make debs fail on debian without systemd enabled" class="issue-link" data-issue-key="LU-11575"><del>LU-11575</del></a> build: install systemd stuff only for debian with systemd enabled</tt>" could possibly be causing the test failures itself. It is more likely that this is being seen on this patch because it is one of the few/only patches that is running Ubuntu client testing. That probably implies that we should have a regular Ubuntu client test session, at least as part of "<tt>full</tt>", or possibly an optional session as part of regular review testing if there are test nodes available.</p> <p>Yeah, agreed to Andreas.</p> <p>Patch <a href="https://review.whamcloud.com/36405/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36405/</a> (<a href="https://jira.whamcloud.com/browse/LU-11575" title="make debs fail on debian without systemd enabled" class="issue-link" data-issue-key="LU-11575"><del>LU-11575</del></a>) is only systemd unit stuff, there's no core change in it. I think these error occurred because the testing clients are Ubuntu, Ubuntu testing sessions are really limited on our normal review, especially 18.04.</p> <p>Gu, any update on this ticket?</p> <p>This failure is affecting patch review testing on Lustre b2_12 branch.</p> <p>What is interesting is that of the 31 failures of sanity.sh test_56ra in the past 4 weeks, 29 of them are on b2_12 (about 30% failure rate on that branch), and the one on master is an interop test with master and b2_12.</p> <p>On master the patch is skipping the test for MDS &lt; 2.12.58, but on b2_12 it isn't doing that. However, the presence of the test on b2_12 implies that the LSOM functionality was backported and <em>should</em> be working.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38021" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38021</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13017" title="sanity test 56ra fails with &#39; &#39;/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity&#39; should send 12 glimpse RPCs to OST&#39;" class="issue-link" data-issue-key="LU-13017"><del>LU-13017</del></a> tests: clean up sanity 56ra add debugging<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 8193ecaa7e199d2dc237a4e318158a7b33d425f3</p> <p>It looks like the first failure of test_56ra on b2_12 with the "<tt>'/usr/bin/lfs find -size 5 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST</tt>" message was on 2019-11-21, which is the same day that patch <a href="https://review.whamcloud.com/36553" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36553</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11367" title="integrate LSOM with lfs find" class="issue-link" data-issue-key="LU-11367"><del>LU-11367</del></a> som: integrate LSOM with lfs find</tt>" landed on that branch. This was confused by some earlier cases of <a href="https://jira.whamcloud.com/browse/LU-13148" title="Interop: sanity test 56ra fails with &#39;&#39;/usr/bin/lfs find -size 0 -type f -lazy /mnt/lustre/d56ra.sanity&#39; should not send glimpse RPCs to OST&#39;" class="issue-link" data-issue-key="LU-13148"><del>LU-13148</del></a> causing test_56ra to also fail on b2_12, but these were caused by interop testing between 2.12.2 and 2.12.58.</p> <p>Patch 38021 has recently landed to b2_12, and most/all of the failures on master are due to interop tests. If this problem is no longer seen on master/b2_12 in a week then this can be closed.</p> <p>I know this is "resolved" so maybe no-one will see this, but I'll report anyway.</p> <p>I got this failure reliably in my testing.  The test is sensitive to timing and performance.</p> <p>I build my kernels with lots of debugging options enabled, and there is a significant performance penalty - but I like the benefits.</p> <p>If it takes more than 1 second from the start of one "lfs find" to the start of the next, failures are quite likely.</p> <p>While the 'lfs find" is calling 'stat' on each file, a background "stat-ahead" thread is doing a similar thing.</p> <p>If "find" gets in before "stat-ahead", the stat-ahead will skip the RPC.  If the <b>previous</b> find did a 'stat' within the last 1 second, "stat-ahead" will again skip the RPC.  But if the previous stat was more than 1 second ago, and the current find hasn't got up to the file yet, the stat-ahead <b>will</b> do an RPC.  Then the find will do its own RPC, resulting in extra RPCs.</p> <p>It seems strange that a "stat()" call will do an RPC even if there was a very recent stat-ahead RPC.  What is the point of stat-ahead in that case?</p> <p> </p> <p>Seems like a bug in statahead?</p> <p>Maybe it's a bug...</p> <p>It appears to be a problem when the 'stat()' system call happens <b>after</b> the stat-ahead, and <b>before</b> the statahead receives a reply.  During that time the lock that was created and requested is not attached to the (inode's) resource, so it isn't found when the 'stat' attempts to find a lock, so instead it creates a new lock.  As both locks are PR they don't conflict and you get two locks on the one resource, both holding size/timestamp info.</p> <p>If the 'stat()' happens <b>before</b> the stat-ahead, the whole operation is under a semaphore so the stat-ahead aborts as it cannot get the lock.  If the stat() happens <b>after</b> the statahead gets a reply, the lock will be attached to the resource, and the stat() will find and re-use it.</p> <p>Maybe we could add a "pending" list to the ldlm_resource, and add locks there that have been requested from the server, but that haven't had a reply yet.  That would allow us to check if a stat-ahead was pending.</p> <p>I tried adding a 'statahead-pending' flag to the inode, but the lock grant doesn't get back to the osc layer - it just sits in the ldlm waiting to be requested again.</p> <p>It is actually the 'agl' thread within the statahead module that is causing the problem.<br/> And, of course,  it is the "mdc" layer that the lock-grant would need to get to, not the "osc" layer.</p> <p>I think the best way forward is to disable agl while the rest is running.</p> <p>This will have the added advantage of testing the "disable agl" code, which is currently buggy</p> <p> </p> <p>Neil Brown (neilb@suse.de) uploaded a new patch: <a href="https://review.whamcloud.com/39667" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39667</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13017" title="sanity test 56ra fails with &#39; &#39;/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity&#39; should send 12 glimpse RPCs to OST&#39;" class="issue-link" data-issue-key="LU-13017"><del>LU-13017</del></a> tests: disable statahead_agl for test_56ra<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d6febfd38d409af0b42291a94ee007acc7f06e23</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39667/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39667/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13017" title="sanity test 56ra fails with &#39; &#39;/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity&#39; should send 12 glimpse RPCs to OST&#39;" class="issue-link" data-issue-key="LU-13017"><del>LU-13017</del></a> tests: disable statahead_agl for sanity test_56ra<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3e04c4f0757c228fc9a1967617e12f3e73e8ffaf</p> Duplicate LU-13378 Related LU-12378 LU-13148 Development Rank 1|i00pzr: Rank (Obsolete) 9223372036854775807 Severity