https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13416 Lustre <p>IAM tables uses a zero copy update for files as similar as ldiskfs directories does.<br/> osd-ldiskfs staring from </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># git describe 67076c3c7e2b11023b943db2f5031d9b9a11329c v2_2_50_0-22-g67076c3 </pre> </div></div> <p>does same. But it's not a safe without set a LDISKFS_INODE_JOURNAL_DATA to inodes.<br/> (thanks bzzz for tip).<br/> Otherwise metadata blocks can be reused before journal checkpoint without corresponded revoke records. It caused a valid file data will replaced with stale journaled data.<br/> from blk trace perspective it shown</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> mdt_io01_025-32148 [003] 4161.223760: block_bio_queue: 9,65 W 12075997800 + 8 [mdt_io01_025] mdt_io01_019-31765 [003] 4163.374449: block_bio_queue: 9,65 W 12075997800 + 8 [mdt_io01_019] mdt_io01_000-12006 [014] 4165.256635: block_bio_queue: 9,65 W 12075997800 + 8 [mdt_io01_000] mdt_io01_019-31765 [004] 4167.030265: block_bio_queue: 9,65 W 12075997800 + 8 [mdt_io01_019] </pre> </div></div> <p>but this info is committed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000001:00080000:9.0:1585615546.198190:0:11825:0:(tgt_lastrcvd.c:902:tgt_cb_last_committed()) snx11281-MDT0000: transno 4522600752066 is committed 00000001:00080000:9.0:1585615546.198196:0:11825:0:(tgt_lastrcvd.c:902:tgt_cb_last_committed()) snx11281-MDT0000: transno 4522600752064 is committed </pre> </div></div> <p>but after crash, journal records is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Commit time 1585612866.905807896 FS block 1509499725 logged at journal block 1370 (flags 0x2) Found expected sequence 86453863, type 2 (commit block) at block 1382 Commit time 1585612871.80796396 Found expected sequence 86453864, type 2 (commit block) at block 1395 Commit time 1585612871.147796211 FS block 1509499725 logged at journal block 1408 (flags 0x2) Found expected sequence 86453865, type 2 (commit block) at block 1414 Commit time 1585612872.386792798 FS block 1509499725 logged at journal block 1427 (flags 0x2) Found expected sequence 86453866, type 2 (commit block) at block 1438 Commit time 1585612876.763804361 Found expected sequence 86453867, type 2 (commit block) at block 1451 Commit time 1585612876.834804666 FS block 1509499725 logged at journal block 1464 (flags 0x2) Found expected sequence 86453868, type 2 (commit block) at block 1471 </pre> </div></div> <p>and none revoke records.</p> Any Lustre 2.x affected. LU-13416

Data corruption during IOR testing with DoM files and hard failover

Bug Blocker Resolved Fixed Alexey Lyashkov Alexey Lyashkov Mon, 6 Apr 2020 16:20:41 +0000 Wed, 23 Dec 2020 09:46:14 +0000 Wed, 20 May 2020 13:36:03 +0000 Lustre 2.14.0 Lustre 2.12.5 0 9 <p>Is the solution here just to set <tt>LDISKFS_JOURNAL_DATA_FL</tt> on IAM files at create/open time (if unset)? I know we can't normally change the "<tt>+j</tt>" flag on an inode without flushing the journal, but this is handled properly via <tt>ldiskfs_ioctl_setflags()</tt> and will only change the flag once per IAM file (normally at mount) so it is not a significant performance concern.</p> <p>Andreas,<br/> solution is simple, but realization is too hard, as OSD don't know is this object is under journaling or not.<br/> In most cases we can check a fid sequences, but sometimes not.</p> <p>'out' protocol is problem, and Wangdi version for DNE2 llogs.<br/> for first case, we don't have an 'open' - just a locate an object and start to write to it.<br/> second case (probably it's different side of first) - it's llogs who uses an normal fid (it can be fixed - but it need to change an object type to create).</p> <p>first case is primary problem - as OUT processing don't have something like 'open' command.</p> <p>PS. lustre large EA is affected also - ext4 version have an exclusion in code, which force to add a revoke records in EA inode destroyed.</p> <p>I'd think that setting this flag in osd_write() and when IAM is being modified should be enough?</p> <p>Alex,</p> <p>I think it's not enough. this flags affects just for objects in memory, but what will be under high memory load? object will flushed from cache and inode is released without unlink. at next time, object allocated again and passed to unlink path. It mean no flag while unlink so .. bug will exist.<br/> In case on journal size 1G and more - this mean up 1000s before journal checkpoint, in comparison to commit which is more often. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=shadow" class="user-hover" rel="shadow">shadow</a> this flag is part of on-disk inode like INODE_EXTENTS, for example.</p> <p>I understand, but inode dirty to have write this flag to disk on each write is too expensive. <br/> Otherwise this flag will don't flushed to disk in case file updates (same size don't want to have mark_inode_dirty) - which is common case for already existent system. </p> <p>PS. i checked a last logs. System had a 4100s up and none checkpoint process was processed and journal is ~20% full. </p> <p>this is to be done once, then mark_inode_dirty() can be skipped.</p> <p>&gt; this is to be done once, then mark_inode_dirty() can be skipped.</p> <p>anyway this change need to be checked with performance evaluation, once DNE2 have so much llog operations.</p> <p>buffer is kept in cache until it's checkpointed, so I don't see why storing the flag in buffer (with mark_inode_dirty()) won't work.</p> <blockquote> <p>anyway this change need to be checked with performance evaluation, once DNE2 have so much llog operations.</p></blockquote> <p>sure, though I'd expect that to be barely visible - any <em>first</em> write to a llog file would set the flag and given llog file is just created, inode is to be modified <em>anyway</em> to get it's block(s). so it could be optimized even with something like the following in osd_write():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (LDISKFS_INODE_JOURNAL_DATA not set) { set LDISKFS_INODE_JOURNAL_DATA; <span class="code-keyword">if</span> (inode-&gt;i_size != 0) mark_inode_dirty(); } </pre> </div></div> <p>the first condition can be wrapped with unlikely(), I guess.</p> <p>Alexey Lyashkov (alexey.lyashkov@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38281" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38281</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13416" title="Data corruption during IOR testing with DoM files and hard failover" class="issue-link" data-issue-key="LU-13416"><del>LU-13416</del></a> ldiskfs: don't corrupt data on journal replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 94b7608c78c85d1d79fec8196ba0fecc9538ab78</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38281/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38281/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13416" title="Data corruption during IOR testing with DoM files and hard failover" class="issue-link" data-issue-key="LU-13416"><del>LU-13416</del></a> ldiskfs: don't corrupt data on journal replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a23aac2219047cb04ed1fa555f31fa39e5c499dc</p> <p>Landed for 2.14</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38705" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38705</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13416" title="Data corruption during IOR testing with DoM files and hard failover" class="issue-link" data-issue-key="LU-13416"><del>LU-13416</del></a> ldiskfs: don't corrupt data on journal replay<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 77c1f307df4a3c068ec45a4948350bc55112e151</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38705/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38705/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13416" title="Data corruption during IOR testing with DoM files and hard failover" class="issue-link" data-issue-key="LU-13416"><del>LU-13416</del></a> ldiskfs: don't corrupt data on journal replay<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 76b1050a56385cf8ddea47c9fea12eec21478601</p> Related LU-14267 Development Rank 1|i00x2v: Rank (Obsolete) 9223372036854775807 Severity