https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13899 Lustre <p>Seeing data corruption on two IOR jobs while running lnet failover test with " Regression write/verify DNE2 PFL". Both the failures showup data from immediate previous run. Likely cache related issue. fsync related warnings are also logged.</p> <p>CL_IOR_all_wr_20iter_1666K_rand</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> access = file-per-process pattern = strided (33 segments) ordering in a file = random offsets ordering inter file=constant task offsets = 1 clients = 48 (4 per node) repetitions = 20 xfersize = 1.63 MiB blocksize = 27.66 MiB aggregate filesize = 42.78 GiB read 1316.54 28322 1666.00 0.001729 33.27 0.010429 33.28 16 XXCEL Using Time Stamp 1594461626 (0x5f098dba) <span class="code-keyword">for</span> Data Signature Commencing write performance test. Sat Jul 11 05:00:26 2020read 1852.87 28322 1666.00 0.001450 23.64 0.004490 23.64 17 XXCEL Using Time Stamp 1594461701 (0x5f098e05) <span class="code-keyword">for</span> Data Signature Commencing write performance test. Sat Jul 11 05:01:41 2020WARNING: cannot perform fsync on file. WARNING: cannot perform fsync on file. write 197.50 28322 1666.00 0.001447 221.82 0.421875 221.82 18 XXCEL Verifying contents of the file(s) just written. Sat Jul 11 05:05:23 2020 [6] At transfer buffer #3, index #0 (file <span class="code-object">byte</span> offset 235425792): [6] Expected: 0x0000000e5f098e05 [6] Actual: 0x0000000e5f098dba [6] At transfer buffer #3, index #2 (file <span class="code-object">byte</span> offset 235425808): [6] Expected: 0x0000000e5f098e05 [6] Actual: 0x0000000e5f098dbaIOR job: -a POSIX -i 20 -w -r -W -t 1666K -b 28322K -C -e -k -vv -E -F -q -s 33 -x -z File in question:/lus/snx11205/ostest.vers/alsorun.20200711032404.2543.pollux-p4/CL_IOR_all_wr_20iter_1666K_rand.2.9t3jzC.1594459731/CL_IOR_all_wr_20iter_1666K_rand/IORfile.00000004 </pre> </div></div> LU-13899

IOR data corruption detected during automated lnet fofb testing

Bug Critical Resolved Fixed Alexander Zarochentsev Alexander Zarochentsev Mon, 10 Aug 2020 17:49:48 +0000 Tue, 1 Sep 2020 04:55:45 +0000 Tue, 1 Sep 2020 04:55:45 +0000 Lustre 2.14.0 0 6 <p>Further debugging showed that<br/> 1. failed fsync() was due to a AS_EIO flag set for the address space, and 492 pages with PG_error flag set:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash-7.2.8&gt; kmem -p | grep ffff8803f2c10c50|awk '{print $7}'|sort|uniq -c 491 error,referenced,uptodate,lru,private 1 error,uptodate,lru,active,private 232875 referenced,uptodate,lru,private 11 referenced,uptodate,private 279 uptodate,lru,active,private crash-7.2.8&gt; </pre> </div></div> <p>2. A debug patch gave us error code -ESTALE: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18282:0:3564:0:(vvp_page.c:304:vvp_page_completion_write()) page@ffff88017556f200[2 ffff8801f8663 0e8 0 1 (null)] 00000020:00000001:28.0:1595864091.118282:0:3566:0:(cl_page.c:840:cl_req_type_state()) Process leaving (rc=2 : 2 : 2) 00000080:00008000:25.0:1595864091.118283:0:3564:0:(vvp_page.c:304:vvp_page_completion_write()) completing WRITE with -116 00000020:00000001:28.0:1595864091.118283:0:3566:0:(cl_page.c:925:cl_page_completion()) Process entered 00000080:00040000:25.0:1595864091.118284:0:3564:0:(vvp_page.c:246:vvp_vmpage_error()) LBUG 00000020:00000001:28.0:1595864091.118285:0:3566:0:(cl_page.c:926:cl_page_completion()) page@ffff8801ff17fc00[2 ffff8801f86630e8 2 1 (null)] 00000020:00000001:28.0:1595864091.118286:0:3566:0:(cl_page.c:926:cl_page_completion()) 1 0 00000020:00000001:28.0:1595864091.118287:0:3566:0:(cl_page.c:341:cl_page_state_set0()) Process entered 00000020:00000001:28.0:1595864091.118288:0:3566:0:(cl_page.c:344:cl_page_state_set0()) page@ffff8801ff17fc00[2 ffff8801f86630e8 2 1 (null)] </pre> </div></div> <p>3. and finally, the -ESTALE comes from (process_req_last_xid):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000200:15.0:1596013191.408113:0:11072:0:(service.c:2233:ptlrpc_server_handle_request()) got req 1673498761985920 00000020:00000001:15.0:1596013191.408114:0:11072:0:(tgt_handler.c:706:tgt_request_handle()) Process entered 00000020:00000001:15.0:1596013191.408114:0:11072:0:(tgt_handler.c:637:process_req_last_xid()) Process entered 00000020:00000001:15.0:1596013191.408115:0:11072:0:(tgt_handler.c:679:process_req_last_xid()) Process leaving via out (rc=18446744073709551500 : -116 : 0xffffffffffffff8c) 00000020:00000001:15.0:1596013191.408116:0:11072:0:(tgt_handler.c:691:process_req_last_xid()) Process leaving (rc=18446744073709551500 : -116 : ffffffffffffff8c) </pre> </div></div> <p>It appears the -ESTALE is not handled properly at client.</p> <p>Alexander Zarochentsev (alexander.zarochentsev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39612" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39612</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13899" title="IOR data corruption detected during automated lnet fofb testing" class="issue-link" data-issue-key="LU-13899"><del>LU-13899</del></a> tgt: drop old epoch request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2af903484430def99448b0e0ba8685aec720dda6</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=zam" class="user-hover" rel="zam">zam</a>, shouldn't this data integrity issue carry a higher priority? I think we ought to target landing for 2.14.0.</p> <p>Hello,<br/> Is a backport planned for b2_12?</p> <p>&gt; shouldn't this data integrity issue carry a higher priority? I think we ought to target landing for 2.14.0.<br/> agreed</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39612/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39612/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13899" title="IOR data corruption detected during automated lnet fofb testing" class="issue-link" data-issue-key="LU-13899"><del>LU-13899</del></a> tgt: drop old epoch request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4f192768293364c65411015de7531f62fdfb754c</p> <p>Landed for 2.14</p> Development Rank 1|i017kv: Rank (Obsolete) 9223372036854775807 Severity