https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13951 Lustre <p>Hello! On Fir storage, we're tracking time when a file's content was last modified. Not the mtime, but the effective time when the file's content had changed. Indeed, we prefer not to rely on the file's mtime which is easily modified by touch.</p> <p>To mitigate this, we currently record files' data_version in Robinhood DB along with a timestamp (max of ctime and mtime). Our purge policy is based on data_version, and if it hasn't changed for 3 months, that means the content has not changed and we purge the file. While this solution works for us, it's not lightweight and it would be easier for us if Lustre could make the last time data_version was updated in Lustre accessible via the liblustreapi.</p> <p>I don't know how difficult it would be to implement this feature, but this is something that could be useful. Thanks!</p> CentOS 7.6 LU-13951

How to get the timestamp of a file data_version's last change

Improvement Minor Open Unresolved Peter Jones Stephane Thiell Mon, 7 Sep 2020 22:13:14 +0000 Tue, 8 Sep 2020 16:00:40 +0000 Lustre 2.12.5 0 4 <p>Hi Stephane, I was thinking about how this might be achieved without adding too much overhead. My understanding is that the concern is primarily about "touch" moving the timestamp forward without actually modifying the file data, and the overhead relates to <b>storing</b> the <tt>data_version</tt> value in RBH? Otherwise, if the <tt>data_version</tt> is stored in RBH, and checked periodically (e.g. at most once a day) when files appear in the Changelog, it isn't clear that storing a separate timestamp in Lustre just for this purpose.</p> <p>One idea that I had was that if "touch" is moving the mtime <em>forward</em>, then this could be stored only on the MDT (which is enough that "<tt>stat()</tt>" will return the correct value), and the OST mtimes do not need to be updated in the setattr case. For mtimes being moved <em>backward</em> the OST mtimes also need to be updated, but I don't think that is a concern. This would allow the OST object mtimes to be used as an actual "last data modified" time, if there was an interface to return those timestamps to userspace on the client.</p> <p>For the userspace interface, there is already <tt>IOC_MDC_GETFILEINFO</tt> that retrieves the layout and the <tt>stat</tt> (or <tt>statx</tt>) data, but the kernel side specifically <b>avoids</b> getting attributes from the OSTs if it can, to minimize overhead. Having a companion interface that <b>only</b> gets the OST attributes would be possible, possibly as a companion/extension of <tt>LL_IOC_DATA_VERSION</tt> would be possible. This ioctl is declared properly:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">enum</span> ioc_data_version_flags { LL_DV_RD_FLUSH = (1 &lt;&lt; 0), <span class="code-comment">/* Flush dirty pages from clients */</span> LL_DV_WR_FLUSH = (1 &lt;&lt; 1), <span class="code-comment">/* Flush all caching pages from clients */</span> LL_DV_TIMESTAMP = (1 &lt;&lt; 2), <span class="code-comment">/* Last data_version update time */</span> }; struct ioc_data_version { __u64 idv_version; __u32 idv_layout_version; <span class="code-comment">/* FLR: layout version <span class="code-keyword">for</span> OST objects */</span> __u32 idv_flags; <span class="code-comment">/* <span class="code-keyword">enum</span> ioc_data_version_flags */</span> }; #define LL_IOC_DATA_VERSION _IOR(<span class="code-quote">'f'</span>, 218, struct ioc_data_version) struct ioc_data_version_v2 { __u64 idv_version; __u32 idv_layout_version; <span class="code-comment">/* FLR: layout version <span class="code-keyword">for</span> OST objects */</span> __u32 idv_flags; <span class="code-comment">/* <span class="code-keyword">enum</span> ioc_data_version_flags */</span> __u64 idv_timestamp; <span class="code-comment">/* last data_version modification time */</span> }; #define LL_IOC_DATA_VERSION_V2 _IOR(<span class="code-quote">'f'</span>, 218, struct ioc_data_version_v2) </pre> </div></div> <p>in that if the size of <tt>struct ioc_data_version</tt> is increased to include a <tt>__u64 idv_timestamp</tt> then it would introduce a new ioctl number ("version") for <tt>LL_IOC_DATA_VERSION_V2</tt> that could be used for compatibility/interop detection and fallback. There could also be a new <tt>LL_DV_TIMESTAMP</tt> flag to indicate whether the timestamp is requested or not.</p> <p>Note that this mechanism wouldn't work for DoM files, because there is not a separate mtime stored for the MDT inode, unlike the OST inode mtime which could be used to only reflect the "data" timestamp.</p> <p>That said, I don't think there is anyone on our side that would be working on this in the near future, as we already have our hands full with other projects and the cutoff for 2.14 features has already behind us.</p> Development Rank 1|i01993: Rank (Obsolete) 9223372036854775807