[LU-4388] fsync on client does not cause OST_SYNCs to be issued

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.6.0
• Affects Version/s: Lustre 2.5.0, Lustre 2.6.0
• Labels: MB
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Nathaniel Clark

Description

Issuing an fsync() only causes MDS_SYNC's to be issued.

Testing:
dd with conv=fsync vs. dd with oflag=sync

Comments

[Hans Henrik Happe]

Should this have been fixed after 2.5.0 or is it only 2.6.0? I see the same problem with b2_5 on zfs (fdatasync okay and slow, fsync not touching the disk). I think it is important that fsync works as expected.

[Jodi Levi (Inactive)]

Patch landed to Master.

[Andreas Dilger]

The patch http://review.whamcloud.com/8626 has not landed yet.

[Jodi Levi (Inactive)]

Patch has landed to Master. Please reopen if more work is needed in this ticket.

[Zhenyu Xu]

patch tracking at http://review.whamcloud.com/8684

[Jinshan Xiong (Inactive)]

I see. So this is a misunderstanding of @datasync parameter. This bug may exist in all series of lustre client implementation. Thanks for explanation.

[Andreas Dilger]

Looking at the comments around vfs_fsync() and vfs_fsync_range()
http://lxr.free-electrons.com/source/fs/sync.c#L178

 * vfs_fsync - perform a fsync or fdatasync on a file
 * @file:               file to sync
 * @datasync:           only perform a fdatasync operation
 *
 * Write back data and metadata for @file to disk.  If @datasync is
 * set only metadata needed to access modified file data is written.

It does appear that datasync=1 means sync data and enough metadata to get the data back (e.g. i_size and block allocations) while datasync=0 means sync everything. See also http://man7.org/linux/man-pages/man2/fsync.2.html

fdatasync() is similar to fsync(), but does not flush modified
metadata unless that metadata is needed in order to allow a
subsequent data retrieval to be correctly handled. For example,
changes to st_atime or st_mtime (respectively, time of last access
and time of last modification; see stat(2)) do not require flushing
because they are not necessary for a subsequent data read to be
handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata
flush.

So if datasync=0 then we should always be sending OST_SYNC RPCs, but if datasync=1 then that is only needed if the size is changed or if blocks were allocated. That is a difficult/impossible decision for a client to make, so the OST_SYNC RPC should be sent to the OST with a DATASYNC flag and let the OSD decide. For older branches it probably makes sense to always send OST_SYNC.

[Zhenyu Xu]

A little bit confused about the datasync parameter of fsync(), is it datasync=1 means only sync data while datasync=0 means sync data+metadata?

[Andreas Dilger]

Does this mean there is a bug in out code that it doesn't send an OST_SYNC in case of conv=fsync?

[Zhenyu Xu]

Nathaniel,

Use conv=fdatasync will issue the OST_SYNC.

dd conv=fsync does not set the datasync parameter for fsync(), conv=fdatasync set it.

[Zhenyu Xu]

dd ... conv=fsync issues several ll_writepages() with wbc->sync_mode == WB_SYNC_ALL and it calls
cl_sync_file_range(inode, start, end, CL_FSYNC_LOCAL, ignore_layout);

after write finishes, it calls ll_fsync() with datasync==0

00000080:00000001:0.0:1388123592.960635:0:20741:0:(rw.c:1112:ll_writepages()) Process entered
...
00000080:00200000:0.0:1388123592.960646:0:20741:0:(file.c:2742:cl_sync_file_range()) VFS Op:inode=[0x200000400:0x1:0x0](ffff8800006da138), sync_mode=1
...
00000080:00000001:1.0:1388123592.973789:0:20741:0:(file.c:2778:cl_sync_file_range()) Process leaving (rc=0 : 0 : 0)
00000080:00000001:1.0:1388123592.973791:0:20741:0:(rw.c:1147:ll_writepages()) Process leaving (rc=0 : 0 : 0)
00000080:00000001:1.0:1388123592.973794:0:20741:0:(file.c:2804:ll_fsync()) Process entered
00000080:00200000:1.0:1388123592.973794:0:20741:0:(file.c:2807:ll_fsync()) VFS Op:inode=[0x200000400:0x1:0x0](ffff8800006da138), datasync=0