https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10810 Lustre <p>lseek with SEEK_HOLE and SEEK_DATA are really helpful  and easy to use tools for any usersapce applications like copy and backup. Currently lustre has min support implementation as per the man page which is not that useful however lustre does support fiemap ioctl which can be used for mapping data in the file.<br/> As we already have support for fiemap I guess with some implementation lustre can support SEEK_HOLE and SEEK_DATA flags. I guess having this support will be helpful to deal with sparse files. Any feedback about the implementation will be really helpful.</p> LU-10810

SEEK_HOLE and SEEK_DATA support for lseek

Improvement Minor Resolved Fixed Mikhail Pershin Lokesh N J medium Tue, 13 Mar 2018 12:31:10 +0000 Fri, 17 Mar 2023 19:48:48 +0000 Sat, 7 Nov 2020 14:23:49 +0000 Lustre 2.14.0 0 14 <p>I was originally thinking that <tt>SEEK_HOLE</tt> and <tt>SEEK_DATA</tt> would be best implemented by calling <tt>FIEMAP</tt> internally, but the remapping of <tt>FIEMAP</tt> data to file-offset extents may be quite complex (and getting worse with PFL, composite layouts, FLR, DoM, etc). It probably makes more sense to have an interface which takes the current file offset, maps it to the object offset(s) (via LOV EA), and then calls <tt>SEEK_HOLE/SEEK_DATA</tt> on the OST (or DoM MDT) for each of the objects that cover the current component, and then LOV maps those object offsets back to file offsets and finds the lowest mapped offset of any of the objects.</p> <p>This approach would be relatively straight forward to implement (LOV can already do file&lt;-&gt;object mapping), though some care would be needed when e.g. crossing component extent boundaries.</p> <p>I was checking when <tt>SEEK_HOLE</tt> and <tt>SEEK_DATA</tt> were added to the kernel (originally kernel commit v3.7-rc3-24-gc8c0df241cc2). This was later fixed by kernel commits v4.12-rc2-3-g7d95eddf313c and v4.14-rc3-4-g545052e9e35a, both of which are already included in RHEL7. On ZFS, this functionality was added to ZPL by commit zfs-0.6.1-56-g802e7b5fe by Dongyang.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> goes on to say:</p> <p>John, for the sparse file handling, IMHO the best (and potentially relatively low-effort) implementation of <tt>SEEK_{HOLE,DATA</tt>} would be as described in <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a>. Essentially, it would be "GETATTR with a <tt>SEEK_{HOLE,DATA</tt>} flag" passing the current file offset and returning the start of the next hole/data. This would require very little extra code, and it leverages the <tt>SEEK_{HOLE,DATA</tt>} implementation of the underlying filesystem plus the existing LOV file offset remapping code. For <tt>SEEK_HOLE</tt> it would return the maximum hole offset of all allocated objects overlapping the current file offset or in later components. For <tt>SEEK_DATA</tt>, it would return the minimum data offset of all allocated objects overlapping the current file offset or in later components.</p> <p>The main drawback of using <tt>SEEK_{HOLE,DATA</tt>} is that it needs changes on both the client and server, but avoids the (IMHO considerable) overhead of all-zero block detection. However, we may need to do zero-block detection in the copy code anyway for compatibility and ease of fast deployment, since that is what <tt>cp &#45;&#45;sparse=always</tt> does in the end if <tt>SEEK_{HOLE,DATA</tt>} is not available. It would be possible to have the copytool try <tt>SEEK_{HOLE,DATA</tt>} first, but if the file is known to be sparse (<tt>st_blocks &lt; st_size</tt>) but <tt>SEEK_HOLE</tt> returns <tt>&gt;= st_size</tt> then <tt>--sparse=always</tt> would fall back to zero block detection. If <tt>SEEK_HOLE</tt> returns <tt>&lt; st_size</tt> then this feature is working. Note that this is <b>not</b> safe for files that are currently being accessed, even if <tt>fsync(fd)</tt> is called before <tt>seek(SEEK_HOLE)</tt> because this doesn't exclude other clients from modifying pages in their own cache. That is not an issue with a local filesystem, but is an issue for a distributed filesystem. The client should probably return <tt>-EOPNOTSUPP</tt> or <tt>st_size</tt> if the <tt>SEEK_HOLE</tt> call returns an offset <tt>&lt; st_size</tt> but the client is not currently holding a <tt>PW</tt> DLM lock on that extent (i.e. some other client could be concurrently writing into that hole).</p> <p>Current status on ticket. I am starting to combine all small parts to working code at this week, server code will be first, then client part and tests. I expect to push working code in gerrit at the next week. It is possible to split it for patches on server changes (no tests, just code) and client changes plus tests.</p> <p>Will you need an OBD_CONNECT flag for this?</p> <p>Andreas, yes I use one already.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define OBD_CONNECT2_LSEEK        0x40000ULL <span class="code-comment">/* SEEK_HOLE/DATA RPC support */</span> </pre> </div></div> <p>I didn't check yet if this value is being used by some other patches</p> <p>I choose to use separate OST_SEEK RPC instead of OST_GETATTR because that would make OST_GETATTR sort of ioctl which does one or another things depending on flags, moreover it is not returning ATTR even just separate information. Also this RPC itself looks obsoleted - it is being used now only to get data_version and would require also new MDT handler and new attr flag. Contrary, OST_SEEK is simple and handled by unified TGT handler for both MDT/OFD.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39706" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39706</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> osd: implement lseek method in OSD<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 24983c305b0c64f42e57ae7d91cac81424099c0f</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39707" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39707</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> ptlrpc: introduce OST_SEEK RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 643bbb697e5439bb720acdb5038067b7bec6b83a</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39708" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39708</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> clio: SEEK_HOLE/SEEK_DATA on client side<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c02e19f65667971f8320b20d99cd5db7199b7c8d</p> <p>Things work with these three patches for normal files. ZFS is not working because needed function <tt>dmu_offset_next</tt> is not exported.</p> <p>More work is needed for released files support, considering that tools uses <tt>SEEK_DATA/SEEK_HOLE</tt> prior data copying I suppose we need to restore released file during lseek operation.</p> <p>Also more tests are needed for cases with concurrent access to the file while lseek is being performed.</p> <p>Please link to the GitHubissue and  pull request for exporting{{dmu_offset_next()}} when you have it. It would also be possible to use symbol lookup in the kernel until the kernel exports it. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39706/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39706/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> osd: implement lseek method in OSD<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 947af94817f8eeb5e6108b0b3cde65419b13c8d3</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39894" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39894</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> test: extended debug for lseek test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f397a37df3b2f8f636686edc37b1720f58aa088a</p> <p>coreutils cp added SEEK_HOLE/DATA support earlier this year. See <a href="https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=a6eaee501f6ec0c152abe88640203a64c390993e" class="external-link" target="_blank" rel="nofollow noopener">https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=a6eaee501f6ec0c152abe88640203a64c390993e</a> </p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40502" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40502</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> test: check seek support in tools<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9dc5e733c58bcdd12a987a4b06eb2f4845597dfb</p> <p>Oleg Drokin (green@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40570" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40570</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> tests: Fix calls to lseek_test_425<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14cc826f465679531e352fd3852b115d369f6773</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39707/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39707/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> ptlrpc: introduce OST_SEEK RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6d5fe29066af5f8e40055fd89b285853c363e947</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39708/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39708/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> clio: SEEK_HOLE/SEEK_DATA on client side<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cda353e6efae5013a26aedbe49d8aa6fb8fe456e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40502/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40502/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> test: test lseek support in tools<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d633172a38519aba2585c2a1fdcdd821cb19010c</p> <p>Landed for 2.14</p> <p>Mike, I noticed some test runs are reporting no SEEK_HOLE support for test_430a,430b,430c, even though client and server are running on the current tip of master (v2_13_56-149-ge5c8f6670f):<br/> <a href="https://testing.whamcloud.com/search?horizon=518400&amp;status%5B%5D=SKIP&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=4154d1b3-ab47-47e4-9ecd-ea85afa35353&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?horizon=518400&amp;status%5B%5D=SKIP&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=4154d1b3-ab47-47e4-9ecd-ea85afa35353&amp;source=sub_tests#redirect</a></p> <p>It doesn't look like all test runs are being skipped, but I didn't think this feature is specific to ZFS or ldiskfs, so I'm not sure why the subtests are being skipped.</p> <p>Andreas, I've check these reports and in fact only couple of them are ldiskfs, in these cases there were old servers used, with just no lseek support. All ZFS tests are expected to be skipped. </p> <p>In fact, I was testing recently ZFS 8.5 with exported <tt>dmu_offset_mext()</tt> to check how it behave and have found that there are problems.<br/> ZFS SEEK_HOLE/DATA support is unreliable because it doesn't work if there are dirty data, that cause all test to fail because we don't do full data sync prior each lseek. I was trying to sync data in tests but with no big success. Moreover I don't think test failures are problems, after all we could sync everything just to make test pass but that would not mean ZFS lseek is reliable until ZFS itself will handle dirty data.</p> <p>For now it can report HOLE at offset 1000 (correct) and immediately SEEK_DATA(1000) returns also 1000 (incorrectly) just because there are dirty data and it switches to generic approach (all file is data with hole at the end). I am afraid we cannot rely on ZFS until its option <tt>zfs_dmu_offset_next_sync</tt> is set, which in turn would cause zfs slowdown when SEEK_DATA is performed on it</p> <p>So basically ZFS server should report LSEEK support only if <tt>zfs_dmu_offset_next_sync</tt> is set AND <tt>dmu_offset_next</tt> is exported. And that should be done now I think, before we will start using ZFS 8.5 and find out that lseek is reported as supported but doesn't work as expected</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50325" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50325</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> osd: implement lseek method in OSD<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b21bc7e262aff42aa3fd90966a791de4ad515dd0</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50326" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50326</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> ptlrpc: introduce OST_SEEK RPC<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: ff27b1c2da37c1d910da48738e97f7193f4453ef</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50327" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50327</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> clio: SEEK_HOLE/SEEK_DATA on client side<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a196f0c1aaac7d1e58aa85847256bc5f2e24f1c1</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50328" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50328</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10810" title="SEEK_HOLE and SEEK_DATA support for lseek" class="issue-link" data-issue-key="LU-10810"><del>LU-10810</del></a> test: test lseek support in tools<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 59154a1c86e186c98359b042f96df389f0b91ebe</p> Duplicate LU-13397 Related LU-1941 LU-3606 LU-13155 LU-14143 LU-11621 LU-14217 Development Rank 1|hzzubr: Rank (Obsolete) 9223372036854775807