https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4388 Lustre <p>Issuing an fsync() only causes MDS_SYNC's to be issued.</p> <p>Testing:<br/> dd with conv=fsync vs. dd with oflag=sync</p> LU-4388

fsync on client does not cause OST_SYNCs to be issued

Bug Blocker Resolved Fixed Zhenyu Xu Nathaniel Clark MB Tue, 17 Dec 2013 15:55:28 +0000 Thu, 14 Jun 2018 21:41:37 +0000 Thu, 24 Apr 2014 13:24:14 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.6.0 0 13 <p>Looking at the ll_fsync() code path I see a number of gaps in the handling. This should be passing through the file range for fsync, since it is possible to sync only a range of the file. The Lustre OST protocol has always supported this, and it makes sense to implement it correctly.</p> <p>On the OST side, it looks like ofd_sync() passes start/end correctly, but the new ofd_sync_hdl() does not. The whole callchain for tgt_sync-&gt;dt_object_sync-&gt;osd_object_sync-&gt;do_sync() need to take start and end arguments, and then pass it down to f_op-&gt;fsync() for those newer variants of fsync() that allow specifying a range of the file.</p> <p>I pushed <a href="http://review.whamcloud.com/8626" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8626</a> to fix handling of </p> {start, end} <p> limits for fsync, but this will not fix the core problem of OST_SYNC not being sent at all. Looking at the twisty maze of CLIO that the fsync call passes through, it <em>seems</em> like the fsync() call should generate an OST_SYNC RPC, but I didn't look at actual debug logs to trace the callpath to see where it actually goes wrong.</p> <p>It is probably worthwhile to create a test case that writes 900kB (less than one RPC, so it isn't sent immediately), fsync (via "<tt>multiop -y</tt>"), immediately fail the OST (mark it read-only) and abort recovery when fsync() returns, then remount the client and verify that the data actually made it to disk. There are sanity.sh test_24</p> {a,b} <p>() that test fsync() behaviour in terms of returning an error to the caller on failure, but there isn't a test that verifies fsync() is actually working correctly. Since we've had problems in this area several times (it is not a problem that can be seen easily, since the data is normally written to disk within a few seconds by itself anyway), it probably makes sense to start by writing this test first, and then using it to debug the problem.</p> <p>Note that my patch is not addressing the core issue here, just a problem that I found when I was looking at this bug originally. Someone with more CLIO skill or time to debug needs to look at this problem.</p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Nathaniel,</p> <p>Can you describe the problem in detail? What's the full commands and the respective outputs?</p> <p>dd ... conv=fsync issues several ll_writepages() with wbc-&gt;sync_mode == WB_SYNC_ALL and it calls <br/> cl_sync_file_range(inode, start, end, CL_FSYNC_LOCAL, ignore_layout);</p> <p>after write finishes, it calls ll_fsync() with datasync==0</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:0.0:1388123592.960635:0:20741:0:(rw.c:1112:ll_writepages()) Process entered ... 00000080:00200000:0.0:1388123592.960646:0:20741:0:(file.c:2742:cl_sync_file_range()) VFS Op:inode=[0x200000400:0x1:0x0](ffff8800006da138), sync_mode=1 ... 00000080:00000001:1.0:1388123592.973789:0:20741:0:(file.c:2778:cl_sync_file_range()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:1.0:1388123592.973791:0:20741:0:(rw.c:1147:ll_writepages()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:1.0:1388123592.973794:0:20741:0:(file.c:2804:ll_fsync()) Process entered 00000080:00200000:1.0:1388123592.973794:0:20741:0:(file.c:2807:ll_fsync()) VFS Op:inode=[0x200000400:0x1:0x0](ffff8800006da138), datasync=0 </pre> </div></div> <p>Nathaniel,</p> <p>Use conv=fdatasync will issue the OST_SYNC.</p> <p>dd conv=fsync does not set the datasync parameter for fsync(), conv=fdatasync set it.</p> <p>Does this mean there is a bug in out code that it doesn't send an OST_SYNC in case of conv=fsync?</p> <p>A little bit confused about the datasync parameter of fsync(), is it datasync=1 means only sync data while datasync=0 means sync data+metadata?</p> <p>Looking at the comments around vfs_fsync() and vfs_fsync_range()<br/> <a href="http://lxr.free-electrons.com/source/fs/sync.c#L178" class="external-link" target="_blank" rel="nofollow noopener">http://lxr.free-electrons.com/source/fs/sync.c#L178</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> * vfs_fsync - perform a fsync or fdatasync on a file * @file: file to sync * @datasync: only perform a fdatasync operation * * Write back data and metadata <span class="code-keyword">for</span> @file to disk. If @datasync is * set only metadata needed to access modified file data is written. </pre> </div></div> <p>It does appear that datasync=1 means sync data and enough metadata to get the data back (e.g. i_size and block allocations) while datasync=0 means sync everything. See also <a href="http://man7.org/linux/man-pages/man2/fsync.2.html" class="external-link" target="_blank" rel="nofollow noopener">http://man7.org/linux/man-pages/man2/fsync.2.html</a></p> <blockquote> <p> fdatasync() is similar to fsync(), but does not flush modified<br/> metadata unless that metadata is needed in order to allow a<br/> subsequent data retrieval to be correctly handled. For example,<br/> changes to st_atime or st_mtime (respectively, time of last access<br/> and time of last modification; see stat(2)) do not require flushing<br/> because they are not necessary for a subsequent data read to be<br/> handled correctly. On the other hand, a change to the file size<br/> (st_size, as made by say ftruncate(2)), would require a metadata<br/> flush.</p></blockquote> <p>So if datasync=0 then we should always be sending OST_SYNC RPCs, but if datasync=1 then that is only needed if the size is changed or if blocks were allocated. That is a difficult/impossible decision for a client to make, so the OST_SYNC RPC should be sent to the OST with a DATASYNC flag and let the OSD decide. For older branches it probably makes sense to always send OST_SYNC. </p> <p>I see. So this is a misunderstanding of @datasync parameter. This bug may exist in all series of lustre client implementation. Thanks for explanation.</p> <p>patch tracking at <a href="http://review.whamcloud.com/8684" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8684</a></p> <p>Patch has landed to Master. Please reopen if more work is needed in this ticket.</p> <p>The patch <a href="http://review.whamcloud.com/8626" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8626</a> has not landed yet.</p> <p>Patch landed to Master.</p> <p>Should this have been fixed after 2.5.0 or is it only 2.6.0? I see the same problem with b2_5 on zfs (fdatasync okay and slow, fsync not touching the disk). I think it is important that fsync works as expected.</p> <p>2.6.0 has the patches and 2.5.0 does not.</p> <p>Given the severity (data loss, inconsistent databases, etc..), shouldn't it be included in the maintenance release?</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/14840" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14840</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4388" title="fsync on client does not cause OST_SYNCs to be issued" class="issue-link" data-issue-key="LU-4388"><del>LU-4388</del></a> llite: issue OST_SYNC for fsync()<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 3951a7c7aed155c206043838e4e800c94ac1c692</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/15390" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15390</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4388" title="fsync on client does not cause OST_SYNCs to be issued" class="issue-link" data-issue-key="LU-4388"><del>LU-4388</del></a> test: add sanity test case<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3112c90215374573c987a800c192300817c62200</p> Related Development Rank 1|hzwbiv: Rank (Obsolete) 12048 Severity