https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6768 Lustre <p>In order to test the stability of the Lustre file system under continuous workload and extreme resource usage, I wrote a tool to write data to files continuously unless 'ENOSPC' occurs, then the tool will truncate and delete some old files to free the space and continue to write the files. Before the file is truncated, its content will be verified. The problem is that after running the tool for a while, the content of some file would be wrong.</p> <p>If the data is corrupted, it will fail with:<br/> yaft: main.cpp:81: void check_file_content(const std::string&amp;): Assertion `rbuf.checkAt(pos)' failed.<br/> Aborted</p> <p>You can get the tool at:<br/> <a href="https://github.com/zhang-jingwang/yaft.git" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zhang-jingwang/yaft.git</a></p> Reproduced in a virtual machine using loop device as OSD-ldiskfs disk. LU-6768

Data corruption when write and truncate in parallel in a almost-full file system

Bug Critical Resolved Fixed Alex Zhuravlev Jingwang Zhang Fri, 26 Jun 2015 02:00:22 +0000 Mon, 14 Sep 2015 14:37:27 +0000 Mon, 3 Aug 2015 15:09:42 +0000 Lustre 2.6.0 Lustre 2.8.0 0 8 <p>My analysis about this problem in an email:<br/> During our tests on Lustre, we found a data corruption when we write to files and truncating/deleting other files at the same time. Usually we could find that one block in the file will contain wrong data which seems very likely to be a metadata block (ext4 extents).</p> <p>And after a long time investigation, we finally found out the root cause for the problem. In short, this is due to a race condition between the data IO (Lustre performs them directly to the block device by submit_bio()) and the metadata IO (ldiskfs performs metadata IO to the block device, which might be cached in the block device’s page cache).</p> <p>Lustre would commit data IO from clients by the osd_submit_bio() function in osd-ldiskfs/osd_io.c :</p> <div class="code panel" style="border-style: solid;border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;border-bottom-style: solid;"><b>osd_io.c</b></div><div class="codeContent panelContent"> <pre class="code-java">241 <span class="code-keyword">static</span> void osd_submit_bio(<span class="code-object">int</span> rw, struct bio *bio) 242 { 243 LASSERTF(rw == 0 || rw == 1, <span class="code-quote">"%x\n"</span>, rw); 244 <span class="code-keyword">if</span> (rw == 0) 245 submit_bio(READ, bio); 246 <span class="code-keyword">else</span> 247 submit_bio(WRITE, bio); 248 } </pre> </div></div> <p>However, there might be dirty data in the block’s page cache. It’s rare but it does might happen if things happen as following order:<br/> 1. A file got truncated, and its extent blocks are updated to complete the truncation. So those blocks became dirty.<br/> 2. The file is deleted, so all its metadata blocks are free now.<br/> 3. One of the metadata blocks is used as data block to hold client’s data. It will be updated by osd_submit_bio().<br/> 4. Then the kernel decides to flush the dirty pages, the data block will be overwritten by the dirty metadata and the data is corrupted.</p> <p>So I think this is a problem and I think the right thing to do is to invalidate the corresponding pages in the block device’s page cache before we issue the bio to make sure that there aren’t any dirty pages, which might overwrite our data later. So we propose to make the following change on osd_submit_bio() to invalidate the page cache.</p> <div class="code panel" style="border-style: solid;border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;border-bottom-style: solid;"><b>osd_io.c</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> void osd_submit_bio(<span class="code-object">int</span> rw, struct bio *bio) { struct inode *bdinode = bio-&gt;bi_bdev-&gt;bd_inode; LASSERTF(rw == 0 || rw == 1, <span class="code-quote">"%x\n"</span>, rw); <span class="code-keyword">if</span> (rw == 0) { submit_bio(READ, bio); } <span class="code-keyword">else</span> { loff_t start = bio-&gt;bi_sector &lt;&lt; 9; loff_t endbyte = start + bio-&gt;bi_size - 1; /* Invalidate the page cache in the block device, otherwise * the dirty data in block device's page cache might corrupt * the data we are going to write. */ truncate_pagecache_range(bdinode, start, endbyte); submit_bio(WRITE, bio); } } </pre> </div></div> <p>Alex</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>&gt; 1. A file got truncated, and its extent blocks are updated to complete the truncation. So those blocks became dirty.<br/> &gt; 2. The file is deleted, so all its metadata blocks are free now.<br/> &gt; 3. One of the metadata blocks is used as data block to hold client’s data. It will be updated by osd_submit_bio().</p> <p>(2) is not quite correct, metadata blocks aren't freed immediately, instead they are scheduled for release up on commit.<br/> see ldiskfs_mb_free_blocks() for the details - when metadata != 0.</p> <p>also, we already have calls to unmap_underlying_metadata() which is supposed to do what you suggested.<br/> I'm still looking at the code.</p> <p>what kernel version do you use?</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15593" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15593</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6768" title="Data corruption when write and truncate in parallel in a almost-full file system" class="issue-link" data-issue-key="LU-6768"><del>LU-6768</del></a> osd: unmap reallocated blocks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: facf7f47ced7debdbaf0d814158661857176b4bf</p> <p>Jingwang, would you mind to try <a href="http://review.whamcloud.com/15593" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15593</a> please?</p> <p>Thanks for looking into this.</p> <p>I'm using CentOS 6.5 with kernel version 2.6.32.431.29.2. And I will try the fix and get back to you later.</p> <p>I run the reproducer for 7 hours and it didn't fail after applying the patch, where it will fail in minutes without the patch, so I believe that the problem is fixed.</p> <p>thanks for the report and testing. please, inspect the patch and help to move it forward.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15593/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15593/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6768" title="Data corruption when write and truncate in parallel in a almost-full file system" class="issue-link" data-issue-key="LU-6768"><del>LU-6768</del></a> osd: unmap reallocated blocks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bcef61a80ab4fa6cee847722184738ba4deeb971</p> <p>Landed for 2.8</p> <p>This may help us on <a href="https://jira.whamcloud.com/browse/LU-6925" title="oss buffer cache corruption" class="issue-link" data-issue-key="LU-6925"><del>LU-6925</del></a>. Can we get a b2_5 back port? Thanks!</p> <p>Is the patch needed by server or client, or both?<br/> Looks like a server patch.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15904" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15904</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6768" title="Data corruption when write and truncate in parallel in a almost-full file system" class="issue-link" data-issue-key="LU-6768"><del>LU-6768</del></a> lvfs: unmap reallocated blocks<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: c8448bce0ad13aeb65c48905e080ddb0c536fc91</p> <p>Hi Jay,<br/> The patch is only needed by server.</p> Related LU-6925 Development Rank 1|hzxgpj: Rank (Obsolete) 9223372036854775807 Severity