https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14326 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/393c6362-298e-472f-be95-f23a1892e9df" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/393c6362-298e-472f-be95-f23a1892e9df</a></p> <p>test_fsx failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-209vm6 crashed during sanity-dom test_fsx [14491.172043] LustreError: 813:0:(osc_cache.c:1113:osc_extent_make_ready()) ASSERTION( last_oap_count &gt; 0 ) failed: [14491.173797] LustreError: 813:0:(osc_cache.c:1113:osc_extent_make_ready()) LBUG [14491.174924] Pid: 813, comm: ldlm_bl_06 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 [14491.176409] Call Trace: [14491.176839] [&lt;ffffffffc06e867c&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [14491.177945] [&lt;ffffffffc06e899c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [14491.178941] [&lt;ffffffffc0cdf284&gt;] osc_extent_make_ready+0xb64/0xe50 [osc] [14491.180097] [&lt;ffffffffc0ce228b&gt;] osc_io_unplug0+0xeeb/0x1900 [osc] [14491.181117] [&lt;ffffffffc0ce6930&gt;] osc_cache_writeback_range+0x9a0/0xfd0 [osc] [14491.182254] [&lt;ffffffffc0cd1925&gt;] osc_lock_flush+0x195/0x290 [osc] [14491.183251] [&lt;ffffffffc0cd1df8&gt;] osc_ldlm_blocking_ast+0x2f8/0x3e0 [osc] [14491.184337] [&lt;ffffffffc0ac713a&gt;] ldlm_cancel_callback+0x8a/0x310 [ptlrpc] [14491.185607] [&lt;ffffffffc0ad2dd1&gt;] ldlm_cli_cancel_local+0xd1/0x420 [ptlrpc] [14491.186747] [&lt;ffffffffc0ad8fec&gt;] ldlm_cli_cancel+0x10c/0x550 [ptlrpc] [14491.187822] [&lt;ffffffffc0cd1c7a&gt;] osc_ldlm_blocking_ast+0x17a/0x3e0 [osc] [14491.189143] [&lt;ffffffffc0add6e8&gt;] ldlm_handle_bl_callback+0xc8/0x3e0 [ptlrpc] [14491.190361] [&lt;ffffffffc0addf9f&gt;] ldlm_bl_thread_main+0x59f/0xaa0 [ptlrpc] </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-dom test_fsx - trevis-209vm6 crashed during sanity-dom test_fsx</p> LU-14326

sanity-dom test_fsx: crash in osc_extent_make_ready()

Bug Critical Resolved Fixed Zhenyu Xu Maloo Tue, 12 Jan 2021 06:04:44 +0000 Sat, 23 Jan 2021 10:31:51 +0000 Fri, 22 Jan 2021 20:58:37 +0000 Lustre 2.14.0 Lustre 2.14.0 0 8 <p>This assertion can be reproduced outside of sanity-dom:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>${LUSTRE}/tests/llmount.sh touch /mnt/lustre/f0 fsx -c 50 -p 100 -N 1000 -l 1048576 -S 0 -d -d /mnt/lustre/f0 </pre> </div></div> <p>On my system it takes fewer ~5 runs of fsx.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1456.917867] LustreError: 6997:0:(osc_cache.c:1113:osc_extent_make_ready()) ASSERTION( last_oap_count &gt; 0 ) failed: [ 1456.920595] LustreError: 6997:0:(osc_cache.c:1113:osc_extent_make_ready()) LBUG [ 1456.922471] Pid: 6997, comm: fsx 3.10.0-1062.9.1.el7.x86_64.debug #1 SMP Mon Mar 16 12:44:56 CDT 2020 [ 1456.924801] Call Trace: [ 1456.925471] [&lt;ffffffffc06c2d4c&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 1456.927163] [&lt;ffffffffc06c306c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 1456.928725] [&lt;ffffffffc0e025e4&gt;] osc_extent_make_ready+0xb64/0xe50 [osc] [ 1456.930547] [&lt;ffffffffc0e05603&gt;] osc_io_unplug0+0xee3/0x1910 [osc] [ 1456.932230] [&lt;ffffffffc0e09d10&gt;] osc_cache_writeback_range+0x9a0/0xfd0 [osc] [ 1456.934128] [&lt;ffffffffc0df8745&gt;] osc_io_fsync_start+0x85/0x1a0 [osc] [ 1456.935852] [&lt;ffffffffc08efa10&gt;] cl_io_start+0x70/0x140 [obdclass] [ 1456.937585] [&lt;ffffffffc0e49e67&gt;] lov_io_call.isra.7+0x87/0x140 [lov] [ 1456.939251] [&lt;ffffffffc0e4a026&gt;] lov_io_start+0x56/0x150 [lov] [ 1456.940703] [&lt;ffffffffc08efa10&gt;] cl_io_start+0x70/0x140 [obdclass] [ 1456.942419] [&lt;ffffffffc08f1eff&gt;] cl_io_loop+0x9f/0x200 [obdclass] [ 1456.943941] [&lt;ffffffffc14bd10b&gt;] cl_sync_file_range+0x2db/0x380 [lustre] [ 1456.945905] [&lt;ffffffffc14e081a&gt;] ll_writepages+0x7a/0x200 [lustre] [ 1456.947975] [&lt;ffffffffa1215b04&gt;] do_writepages+0x24/0x50 [ 1456.949314] [&lt;ffffffffa1207e35&gt;] __filemap_fdatawrite_range+0x65/0x80 [ 1456.950776] [&lt;ffffffffa1207f94&gt;] filemap_write_and_wait_range+0x44/0x90 [ 1456.952472] [&lt;ffffffffc14bd240&gt;] ll_fsync+0x90/0x540 [lustre] [ 1456.953904] [&lt;ffffffffa12e5770&gt;] vfs_fsync_range+0x20/0x30 [ 1456.955335] [&lt;ffffffffa124e18e&gt;] SyS_msync+0x1fe/0x250 [ 1456.962318] [&lt;ffffffffa1887d4a&gt;] tracesys+0xa6/0xcc [ 1456.963771] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [ 1456.965134] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Bobijam, could you please take a look. This is being hit pretty regularly since <tt>fallocate()</tt> checking has been fixed in the client, and <tt>fsx</tt> is now properly testing that functionality.</p> <p>Is it possible that this LASSERT() is just wrong for <tt>fallocate()</tt> when there are no data pages being sent in the RPC?</p> <p>It looks like there was a patch <a href="https://review.whamcloud.com/30848" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30848</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10407" title="osc_cache.c:1141:osc_extent_make_ready()) ASSERTION( last_oap_count &gt; 0 ) failed: " class="issue-link" data-issue-key="LU-10407"><del>LU-10407</del></a> osc: update size before queue page</tt>" pushed a couple of years ago and never landed, due to Jinshan giving it a -1. That might give some ideas where to start.</p> <p>This is a major sanity-benchmark crashes source (in fsx) for me.</p> <p>I just also filed <a href="https://jira.whamcloud.com/browse/LU-14333" title="ASSERTION( oinfo-&gt;loi_lvb.lvb_size &gt;= oinfo-&gt;loi_kms ) failed" class="issue-link" data-issue-key="LU-14333"><del>LU-14333</del></a> that seems to be related, it hits if this one did not hit, at a later point in time in cleanup.</p> <p>Oleg reports that the 30848 patch does not solve the fsx test issues being seen.</p> <p>my test shows that #30848 only reduced the hit probability, and I got one hit may sheds some light.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000814 1610683006.953568 write 0x09385c thru 0x098924 (0x0550c9 bytes) 000817 1610683006.954846 fallocate 0x092fd1 thru 0x09bfb3 (0x058fe3 bytes)</pre> </div></div> <p>then the issue got hit with some debug info shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1361.292883] LustreError: 16917:0:(osc_cache.c:1287:osc_refresh_count()) kms 602065(0x92fd1) page offset 622592(0x98000) [ 1361.293036] LustreError: 16917:0:(osc_cache.c:1113:osc_extent_make_ready()) ASSERTION( last_oap_count &gt; 0 ) failed: last_oap_count 0 </pre> </div></div> <p>the file's kms and fallocate offset coincides (0x092fd1), the last sync page looks from the write (0x98000 -&gt; 0x98924).</p> <p><del>Bobi Jam (bobijam@hotmail.com) uploaded a new patch:</del> <a href="https://review.whamcloud.com/41233" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41233</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-14326" title="sanity-dom test_fsx: crash in osc_extent_make_ready()" class="issue-link" data-issue-key="LU-14326"><del>LU-14326</del></a> osc: correctly update size/kms for fallocate</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 3</del><br/> <del>Commit: a202c4dbbd8dc6f763b6672a9fe604a9179f209a</del></p> <p>Ran the fsx test for 1000 times with the patch, no LBUG happened again. </p> <p>please see my comment in the gerrit. it shifted failures to <a href="https://jira.whamcloud.com/browse/LU-14333" title="ASSERTION( oinfo-&gt;loi_lvb.lvb_size &gt;= oinfo-&gt;loi_kms ) failed" class="issue-link" data-issue-key="LU-14333"><del>LU-14333</del></a> style crashes and also now sanityn fsx tests in test_16<span class="error">&#91;abc&#93;</span> are 100% failure because fsx sees wrong size.</p> <p>I think the patch set 3 is a correct fix.</p> <p>I don't think this is fallocate issue. This problem has been around a long time. Its just fallocate really exposes this problem.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/41272" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41272</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14326" title="sanity-dom test_fsx: crash in osc_extent_make_ready()" class="issue-link" data-issue-key="LU-14326"><del>LU-14326</del></a> osc: correctly update size/kms for fallocate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 114140000a298b9ee0e774fc1785fed034961afb</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41272/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41272/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14326" title="sanity-dom test_fsx: crash in osc_extent_make_ready()" class="issue-link" data-issue-key="LU-14326"><del>LU-14326</del></a> osc: correctly update size/kms for fallocate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 43979e4e257e78d3d94ea3f21b65fdebbbdc690f</p> <p>Landed for 2.14</p> Related LU-11463 LU-14286 LU-14333 LU-10407 Development Rank 1|i01j5j: Rank (Obsolete) 9223372036854775807 Severity