https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1442 Lustre <p>A data integrity test run periodically run by our storage group found two occurrences of corrupt files written to Lustre. The original files contain 300 MB of random data. The corrupt copies contain several 4096B regions of zeros aligned on 1MiB boundaries. The two corrupt files were written to the same filesystem from two different login nodes on the same cluster within five minutes of each other. The stripe count is 100.</p> <p>The client application is a parallel ftp client reading data out of our storage archive into Lustre. The test checks for differences between the restored files and the original copies. For a 300MB file it uses 4 threads which issue 4 64MB pwrite()'s and 1 44MB pwrite(). It is possible that the pwrite() gets restarted due to SIGUSR2 from a master process, though we don't know if this occurred in the corrupting case. This test has seen years of widespread use on all of our clusters, and this is the first reported incidence of this type of corruption, so we can characterize the frequency as rare.</p> <p>When I examine an OST object containing a corrupt region, I see there is no block allocated for the corrupt region (in this case, logical block 256 is missing). </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pigs58 /root &gt; debugfs -c -R "dump_extents /O/0/d$((30205348 % 32))/30205348" /dev/sdb debugfs 1.41.12 (17-May-2010) /dev/sdb: catastrophic mode - not reading inode or group bitmaps Level Entries Logical Physical Length Flags 0/ 0 1/ 3 0 - 255 813140480 - 813140735 256 0/ 0 2/ 3 257 - 511 813142528 - 813142782 255 0/ 0 3/ 3 512 - 767 813143040 - 813143295 256 </pre> </div></div> <p>Finally, the following server-side console messages appeared at the same time one of the corrupted files was written, and mention the NID of the implicated client. The consoles of the OSTs containing the corrupt objects were quiet at the time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 17 01:06:08 pigs-mds1 kernel: LustreError: 20418:0:(mdt_recovery.c:1011:mdt_steal_ack_locks()) Resent req xid 1402165306385077 has mismatched opc: new 101 old 0 May 17 01:06:08 pigs-mds1 kernel: Lustre: 20418:0:(mdt_recovery.c:1022:mdt_steal_ack_locks()) Stealing 1 locks from rs ffff880410f62000 x1402165306385077.t125822723745 o0 NID 192.168.114.155@o2ib5 May 17 01:06:08 pigs-mds1 kernel: Lustre: All locks stolen from rs ffff880410f62000 x1402165306385077.t125822723745 o0 NID 192.168.114.155@o2ib5 </pre> </div></div> <a href="https://github.com/chaos/lustre/commits/2.1.1-llnl">https://github.com/chaos/lustre/commits/2.1.1-llnl</a> LU-1442

File corrupt with 1MiB-aligned 4k regions of zeros

Bug Major Closed Won't Fix Jinshan Xiong Ned Bass llnl Fri, 25 May 2012 18:46:41 +0000 Tue, 16 Aug 2016 16:36:46 +0000 Tue, 16 Aug 2016 16:36:46 +0000 Lustre 2.3.0 Lustre 2.1.1 Lustre 2.3.0 Lustre 2.1.3 0 17 <p>The client logs were also quiet at the time of the corruption.</p> <p>I should also mention that the corruption offsets are not at 64MiB boundaries, so it is less likely that the application accidentally wrote a sparse file by miscalculating the offset argument to pwrite(). The first file had corruption as MiB offsets 159, 162, 173, 189. In the second file the MiB offsets were 158 and 190.</p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Would you please upload debug log of MDS and client as much as possible?</p> <p>May 17 01:06:08 pigs-mds1 kernel: LustreError: 20418:0:(mdt_recovery.c:1011:mdt_steal_ack_locks()) Resent req xid 1402165306385077 has mismatched opc: new 101 old 0</p> <p>the operation request from client is abnormal (its opc was 0, which is abnormal), the client opc saved on MDS is stored in target_send_reply()</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>target_send_reply()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* disable reply scheduling <span class="code-keyword">while</span> I'm setting up */</span> rs-&gt;rs_scheduled = 1; rs-&gt;rs_on_net = 1; rs-&gt;rs_xid = req-&gt;rq_xid; rs-&gt;rs_transno = req-&gt;rq_transno; rs-&gt;rs_export = exp; rs-&gt;rs_opc = lustre_msg_get_opc(rs-&gt;rs_msg); </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lustre_msg_get_opc()</b></div><div class="codeContent panelContent"> <pre class="code-java">__u32 lustre_msg_get_opc(struct lustre_msg *msg) { <span class="code-keyword">switch</span> (msg-&gt;lm_magic) { <span class="code-keyword">case</span> LUSTRE_MSG_MAGIC_V2: { struct ptlrpc_body *pb = lustre_msg_ptlrpc_body(msg); <span class="code-keyword">if</span> (!pb) { CERROR(<span class="code-quote">"invalid msg %p: no ptlrpc body!\n"</span>, msg); <span class="code-keyword">return</span> 0; } <span class="code-keyword">return</span> pb-&gt;pb_opc; } <span class="code-keyword">default</span>: CERROR(<span class="code-quote">"incorrect message magic: %08x(msg:%p)\n"</span>, msg-&gt;lm_magic, msg); LBUG(); <span class="code-keyword">return</span> 0; } } </pre> </div></div> <p>Hi,</p> <blockquote><p>Would you please upload debug log of MDS and client as much as possible?</p></blockquote> <p>Unfortunately the MDS was rebooted since this event and the debug log was lost. I dumped the debug logs for both clients involved, but I don't think you'll find anything useful in them. I'll attach them just in case.</p> <p>Ned,<br/> can you please check the client NID that reported this problem and check the Lustre version. If the client is a bit older, it is possible that it hit <a href="https://jira.whamcloud.com/browse/LU-721" title="Parallel writes to same file results in a file of zeroes" class="issue-link" data-issue-key="LU-721"><del>LU-721</del></a>, which was only fixed in Lustre 1.8.7, 2.1.1, and 2.2+. AFAIK, you have 1.8.5 clients still in use.</p> <p>Failing that, is there any chance that the client was evicted from the OST, but the console messages related to client eviction have been turned off? There were a lot of patches to quiet console errors, so I'm wondering if a client eviction could happen without any visible console error messages.</p> <p>The write was never sent to the filesystem (evidenced by the unallocated block), so it is unlikely to be corruption at the underlying block layer or in ldiskfs. It is much more likely that the RPC was not handled at all, which could be caused by eviction, or something like <a href="https://jira.whamcloud.com/browse/LU-721" title="Parallel writes to same file results in a file of zeroes" class="issue-link" data-issue-key="LU-721"><del>LU-721</del></a>.</p> <p>Andreas, the clients that are hitting this are running <a href="https://github.com/chaos/lustre/tree/2.1.1-11chaos" class="external-link" target="_blank" rel="nofollow noopener">2.1.1-11chaos</a>, and the servers are all running <a href="https://github.com/chaos/lustre/tree/2.1.1-4chaos" class="external-link" target="_blank" rel="nofollow noopener">2.1.1-4chaos</a> or newer. So we definitely already have the <a href="https://jira.whamcloud.com/browse/LU-721" title="Parallel writes to same file results in a file of zeroes" class="issue-link" data-issue-key="LU-721"><del>LU-721</del></a> patch that is landed in 2.1.1 on those clients.</p> <p>We have 1.8.5-?chaos on some x86_64 clients (again, all server are 2.1.1+) still, but none of them exhibit this problem. These clients will be upgraded to 2.1.1-*chaos in time. But again, only the 2.1.1+ clients are seeing this bug.</p> <p>We are NOT silencing the eviction messages. We absolutely need those.</p> <p>But there are no messages about eviction or anything else on the clients in question for hours on either side of the problem.</p> <p>We have confirmation from the pftp developers that the parent process will send SIGUSER2 to all four of the children (probably when the first child completes). Here is exactly what we've been told:</p> <p>"Near the end of the transfer, the parent process send SIGUSR2 to all four children. Anywhere from 0 to 3 of the 4 children exit pwrite with a return value of -1 and errno of 4 (interrupted system call). All interrupted pwrites are reissued and then succeed."</p> <p>Andreas,</p> <blockquote> <p>The write was never sent to the filesystem (evidenced by the unallocated block), so it is unlikely to be corruption at the underlying block layer or in ldiskfs. It is much more likely that the RPC was not handled at all, which could be caused by eviction, or something like <a href="https://jira.whamcloud.com/browse/LU-721" title="Parallel writes to same file results in a file of zeroes" class="issue-link" data-issue-key="LU-721"><del>LU-721</del></a>.</p></blockquote> <p>One thing that puzzles me about this explanation is that the size of the hole is only 4k. I would expect the client to have been sending 1MB write RPCs since the pwrite() size was a multiple of 1MB. If the write was never sent or handled, shouldn't the hole be 1MB?</p> <p>This bug may be related to <a href="https://jira.whamcloud.com/browse/LU-1299" title="running truncated executable causes spewing of lock debug messages" class="issue-link" data-issue-key="LU-1299"><del>LU-1299</del></a> where sleeping on a cl_lock is interrupted by the signal, so the lock is wrongly marked as error. As a result, the lock canceling will abort and this causes some dirty pages are not flushed.</p> <p>There is a patch at <a href="http://review.whamcloud.com/2574" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2574</a> can you please try if it can fix your issue?</p> <p>Hi Jinshan,</p> <p>We have patchset 7 of that patch in our current tag. We will update to the latest. However we don't have a reliable reproducer for this bug so it will be difficult to say if it fixes the issue.</p> <p>I've updated the 2.1.1-llnl branch to have patch set 10 from <a href="http://review.whamcloud.com/2574" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2574</a>.</p> <p>ok I will close this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1299" title="running truncated executable causes spewing of lock debug messages" class="issue-link" data-issue-key="LU-1299"><del>LU-1299</del></a> and we can reopen if that is proven to not be the case</p> <p>We had a new occurrence of this bug this morning on a client running patchset 7 of <a href="http://review.whamcloud.com/2574" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2574</a>. Does this change your view that this is related to <a href="https://jira.whamcloud.com/browse/LU-1299" title="running truncated executable causes spewing of lock debug messages" class="issue-link" data-issue-key="LU-1299"><del>LU-1299</del></a>, or do we need patchset 10 to prevent the corruption?</p> <p>The new occurrence again coincided with messages on the MDS like these:</p> <p>Resent req xid ... has mismatched opc: new 101 old 0<br/> Stealing 1 locks from rs ...<br/> All locks stolen from rs ...</p> <p>Hi Ned, it seems this is not related to <a href="https://jira.whamcloud.com/browse/LU-1299" title="running truncated executable causes spewing of lock debug messages" class="issue-link" data-issue-key="LU-1299"><del>LU-1299</del></a>. Did the processes write their portion of file exclusively, i.e. there is no extent lock confliction?</p> <blockquote><p>Did the processes write their portion of file exclusively, i.e. there is no extent lock confliction?</p></blockquote> <p>I'm not sure how to determine the answer to that. The pwrite() calls from one strace of the four involved processes looked like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>6621:pwrite(8, "...", 67108864, 0) = 67108864 6621:--- SIGUSR2 (User defined signal 2) @ 0 (0) --- 6621:pwrite(8, "...", 46137344, 268435456) = 46137344 6622:pwrite(4, "...", 67108864, 67108864) = 67108864 6622:--- SIGUSR2 (User defined signal 2) @ 0 (0) --- 6623:pwrite(4, "...", 67108864, 134217728) = ? ERESTARTSYS (To be restarted) 6623:--- SIGUSR2 (User defined signal 2) @ 0 (0) --- 6623:pwrite(4, "...", 67108864, 134217728) = 67108864 6624:pwrite(4, "...", 67108864, 201326592) = ? ERESTARTSYS (To be restarted) 6624:--- SIGUSR2 (User defined signal 2) @ 0 (0) --- 6624:pwrite(4, "...", 67108864, 201326592) = 67108864 </pre> </div></div> <p>(This was not a corrupting case, just a sample run of the test program.)</p> <p>It may be worth noting that in each occurrence we've had so far, the corrupted regions have all been at offsets between 100-200 MB. Since the test uses a stripe count of 100 that's always in the second stripe of the file.</p> <blockquote><p>Did the processes write their portion of file exclusively, i.e. there is no extent lock confliction?</p></blockquote> <p>The files write to their own ranges in the files, and do not overlap with any of the other writers. Furthermore, the writers are all on one node, so there is no ldlm lock contention with any other lustre clients.</p> <p>It seems like this ticket should be reopened.</p> <p>Jinshan,<br/> I would investigate the RPC opcode issue. It would be possible to add debugging on the client to verify the opcode of the RPC is non-zero when it is put into the replay list, and print a message at that point if the opcode is bad, and also verify the opcode on the client before the RPC is resent. If only the second message triggers, it means some kind of in-memory corruption is happening on the client.</p> <p>What is strange is that there are messages on the MDS, instead of on the OSS where one would expect any errors related to file IO to happen. The only thing I can think of where an MDS error would cause bad data is that some of the client threads are somehow opening the incorrect file and the data is going to the wrong location?</p> <p>I'm working on this issue.</p> <p>Verifying RPC opcode would be the only way to start with. The strange thing is that only one page in the middle was missing, I can't connect this with MDS replay error.</p> <p>Hi Ned, did you notice that when this issue happened, pwrite() kept returning -1 with errno ERESTARTSYS, or sometimes short write occurred?</p> <p>just in case, did you use DIRECT_IO to write the file?</p> <blockquote><p>did you notice that when this issue happened, pwrite() kept returning -1 with errno ERESTARTSYS, or sometimes short write occurred?</p></blockquote> <p>I don't have a trace of the process when the issue happened. All I know is that the parent process will signal its children with SIGUSR2 which will cause pwrite() to return -1 with errno ERESTARTSYS. I've never seen this happen more than once in my tests, but it's possible this pattern will repeat if pwrite() is slow. I don't have the pftp source code to check, but I can ask our HPSS group to look into it.</p> <blockquote><p>just in case, did you use DIRECT_IO to write the file?</p></blockquote> <p>No, strace shows pftp just uses O_WRONLY|O_CREAT.</p> <p>Hi Ned, will you please try this patch: <a href="http://review.whamcloud.com/3194" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3194</a>, this patch may fix corruption issue.</p> <p>After the corruption issue is fixed, I'll start to work on wrong opc issue if it's bothering you guys.</p> <p>Great, we'll give the patch a try. We've had three corruptions in about two months, and we haven't found a way to easily reproduce it. So it may take a few months with no new corruptions to gain some confidence in the fix.</p> <p>When I was trying to reproduce this with fail_loc today I found something new. Actually, though a dirty page will not be added into osc's cache if osc_page_cache_add() is interrupted by signal, the page will still be written back by kernel flush daemon. Saying that, the IO pattern can be exactly what you have seen(a page of gap in block allocation), but data corruption is unexpected. I still need to investigate this problem, but I will focus on if there exists a code path causing dirty page discarded.</p> <p>I'm pretty sure the issue you have seen is related to the problem I found in the patch, so please apply this patch and do test intensely. Maybe we can find new clues. Thanks,</p> <p>Hi Ned, Do you know how the application detected the data corruption issue? Do they just read the data back on the same client or some operations, for example flush caching pages, were done between the write and verification?</p> <p>Attached test script. No cache flush or other operations are done between write and verification. The entire test is run on the same client. The process is basically:</p> <ol> <li>write random patterns to a file</li> <li>ftp file to archvial storage</li> <li>retrieve copy from archival storage</li> <li>compare copy to original with 'cmp' to check for corruption</li> </ol> <p>Jinshan, I've added <a href="http://review.whamcloud.com/3194" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3194</a> to our branch to include in the next round of testing.</p> <p>Master version of patch has been merged - <a href="http://review.whamcloud.com/#change,3447" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3447</a></p> <p>Hi Chris, how long have you been running the test with this patch?</p> <p>I don't know, any pushes to gerrit that have rebased on this patch will have been run with this patch. I cannot know who has rebased and pushed.</p> <p>Hi Chris Gearing, Sorry I meant to say Christopher Morrone because LLNL is verifying if the patch can fix the data corruption problem. This is a rarely occurred problem so it may take months to verify it.</p> <p>Its been on our test systems and hasn't caused any problems that I am aware of. It is not installed in production yet. It might make it into a production release in a couple of weeks.</p> <p>We've seen the <a href="https://jira.whamcloud.com/browse/LU-1680" title="LBUG cl_lock.c:1949:discard_cb()) (ORI-726)" class="issue-link" data-issue-key="LU-1680"><del>LU-1680</del></a> failures on the orion branch, and just recently pulled this <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> patch into there. We'll keep an eye out for failures on orion when we upgrade to that version.</p> <p>Oh, in other words, I don't know if it fixes the problem because we don't know hot to reproduce it in our test environment. But it hasn't caused any new problems that I know of.</p> <p>Are the messages below evidence of data corruption? We have quite a number of these on our 2.1.2 mds:</p> <p>LustreError: 4869:0:(mdt_recovery.c:1011:mdt_steal_ack_locks()) Resent req xid 1407363869661366 has mismatched opc: new 101 old 0^M<br/> Lustre: 4869:0:(mdt_recovery.c:1022:mdt_steal_ack_locks()) Stealing 1 locks from rs ffff8802a0930000 x1407363869661366.t210874408121 o0 NID 10.151.26.25@o2ib^M<br/> Lustre: 4265:0:(service.c:1865:ptlrpc_handle_rs()) All locks stolen from rs ffff8802a0930000 x1407363869661366.t210874408121 o0 NID 10.151.26.25@o2ib^M</p> <p>Hi Jay Lan,</p> <p>This error messages are not related as data corruption happened on OST and the message showed something wrong with MDT.</p> <p>I'm going to set this issue as fixed otherwise we can't release 2.3.</p> <p>Please reopen this issue if it occurs again.</p> <p>Hi Jinshan, but our console log messages look the same as the server console log messages shown in the Description section of the ticket. Are you suggesting that other problems can produce the same log messages and thus the messages alone not sufficient evidence? Please advise. Thanks!</p> <p>This message may not introduce any problem, probably just too aggressive console output. I think it can be ignored if you didn't see any real problem.</p> <p>Hi Jay Lan, Lu-1717 was just created to address/understand this MDS console error message.</p> <p>Thanks, Jinshan!</p> <p>Hi,</p> <p>We got a similar issue (the written file has a big hole with a lot of zeros) occasionally and caused wrong simulation results.</p> <p>As we have Luster 1.8.8, I wonder if our file system could be affected by this bug.</p> <p>Thanks,<br/> Yusong </p> <p>Our test program detected this type of corruption again, so we should re-open this issue. Client was running 2.1.2-4chaos, which includes the patch "<a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> llite: cleanup if a page failed to add into cache".</p> <p><a href="https://github.com/chaos/lustre/commits/2.1.2-4chaos" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/commits/2.1.2-4chaos</a></p> <p>Note that this test runs regularly on our production systems, so this bug is very rare to reproduce.</p> <p>Looking at the on-disk data, it is again logical block 256 that is unallocated in the OST object.</p> <p>Ned Bass,</p> <p>Would you mind checking whether the stripe patterns are always the same for the error cases? Also please collect and upload client's rpc_stats of clients info (e.g. /proc/fs/lustre/osc/lustre-OST0000-osc-ffff88003de72800/rpc_stats data).</p> <p>We'd like to know whether clients have always written 1M RPC to OST w/o partial write, so that we'd know whether client's issue or OST's.</p> <p>The stripe patterns are not always the same. The first case involved a 300 MB file with a stripe count of 100. This most recent case involved a 10 GB file with a stripe count of 2. The only consistent pattern so far is that it always seems to be logical block 256 missing from an OST object.</p> <p>I'll check the rpc_stats tomorrow.</p> <p>I can't think of what kind of problem can cause this symptom, there is even no lock contention. One more question is: is the client ppc64?</p> <p>No, both client and server are x86_64.</p> <p>The rpc_stats for the involved OST looked like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 3 0 0 2: 0 0 0 | 0 0 0 4: 2 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 1 0 0 64: 0 0 0 | 3 0 0 128: 0 0 0 | 3 0 0 256: 4771 99 100 | 4768 99 100 read write offset rpcs % cum % | rpcs % cum % 1: 4772 99 99 | 4760 99 99 2: 0 0 99 | 4 0 99 4: 2 0 99 | 0 0 99 8: 0 0 99 | 0 0 99 16: 0 0 99 | 0 0 99 32: 0 0 99 | 1 0 99 64: 1 0 100 | 6 0 99 128: 0 0 100 | 7 0 100 </pre> </div></div> <p>Have you ever run other jobs in this cluster because I saw some non 1M RPCs in the roc_stats?</p> <p>From the symptom, I guess somehow the 1st page of a stripe was not included in the RPC. If that is true, then that RPC must have only 255 pages and we should see a tally in the stats. Of course this way only works if that job is the only one in the cluster.</p> <p>We'll make a debug patch for this purpose.</p> <p>Hi Jinshan,</p> <p>This node has been up for about 70 days and has run many other jobs, so we can't say for sure whether the missing page is reflected in the stats.</p> <p>I like the idea of a debug patch to catch this case. We discussed that approach during our last meeting but I couldn't easily figure out where to put the debugging.</p> <p>I'm lowering the prio to major and will continue working on this after getting new clues.</p> <p>Moved unrelated comments to new bug <a href="https://jira.whamcloud.com/browse/LU-3219" title="FIEMAP does not sync data or return cached pages" class="issue-link" data-issue-key="LU-3219"><del>LU-3219</del></a>.</p> <p>.</p> <p>Reclosing this ticket. The new issue mentioned is tracked in <a href="https://jira.whamcloud.com/browse/LU-3219" title="FIEMAP does not sync data or return cached pages" class="issue-link" data-issue-key="LU-3219"><del>LU-3219</del></a>.</p> <p>Jodi, afaik this issue is not resolved, just rare. Are there still any plans to make a debug patch?</p> <p>Sorry for the confusion Ned - things just got a little muddled due to two somewhat similar issues becoming intertwined. Yes, this is still on Jinshan's plate to dig into further at some point.</p> <p>Old ticket for unsupported version</p> Duplicate LU-1680 Related LU-1703 LU-1458 Development End date Fri, 27 Jun 2014 18:46:41 +0000 Rank 1|hzv633: Rank (Obsolete) 4520 Severity Start date Fri, 25 May 2012 18:46:41 +0000