https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1158 Lustre <p>The current Lustre network protocol has support for a 64-bit timestamp of seconds, but does not have a field for passing the nanosecond timestamp from clients to servers and back again.</p> <p>It would be relatively straight-forward to put 3x <tt>&#95;&#95;u32</tt> nanosecond timestamps in the reserved fields in <tt>struct obdo</tt> and <tt>struct mdt&#95;body</tt>. These fields are currently always initialized to 0, so there wouldn't even need to be a protocol change or feature to begin using these fields for nanoseconds - just copy them in/out of the RPC structures, and old clients/servers will just store 0 there, and ignore any nanosecond timestamps that are sent to them (no differently than they do today).</p> <p>It is more complex to add the nanosecond timestamps to struct ost_lvb, which is most commonly used for glimpse locks (stat) on OST objects. This will require a structure change to fit the extra 3x <tt>&#95;&#95;u32</tt> nanosecond timestamps into ost_lvb, which may require a protocol change. It <em>may</em> be possible if this structure is passed in a separate ptlrpc message buffer that the larger size will be ignored by older clients, which would avoid the need for additional complexity for interoperability.</p> LU-1158

nanosecond timestamp support for Lustre

Improvement Major Reopened Unresolved Feng Lei Andreas Dilger Thu, 1 Mar 2012 14:10:22 +0000 Sat, 3 Feb 2024 22:11:46 +0000 Lustre 2.4.0 0 16 <p>Johann, you were just working on changing the LDLM LVB structure to allow different LVB data to be passed for quota. Does it seem practical to include the modifications necessary to increase ost_lvb in your changes, or is your change not so intrusive (i.e. using a smaller struct than ost_lvb) that adding this ability would increase the complexity of your patch unnecessarily?</p> <p>Right, i've already made changes to the LDLM LVB on the orion_quota branch, although i have not increased the size of it yet.<br/> Yes, i think it makes sense to include the modifications to increase the ost_lvb in the same patch.</p> <p>Patch tracking at <a href="http://review.whamcloud.com/3266" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3266</a></p> <p><a href="https://jira.whamcloud.com/browse/LU-1158" title="nanosecond timestamp support for Lustre" class="issue-link" data-issue-key="LU-1158">LU-1158</a> general: support for nanosecond m/a/ctimes in Lustre</p> <p>I wanted to start a discussion here about the nanosecond times usage of the reserved fields in mdt_rec_reint. I wasn't aware of it when this project started, but I see during patch inspection that the <tt>rr&#95;{m,a,c}time&#95;ns</tt> fields consume the last 3 fields in that template struct. In most of the instantiations (mdt_rec_link, mdt_rec_setxattr, mdt_rec_rename, mdt_rec_unlink) there are plenty of other padding fields that could be used if we need to add something to the wire protocol. In a few of them (mdt_rec_create, mdt_rec_setattr) there are only 2 or 3 unused fields left, and in the case of mdt_rec_setattr there are no 64-bit fields at all.</p> <p>The question is what should we do at this point? Should we hope that we don't need to add too many new fields to these structures? Should we not add the atime_ns field, and just stick with one second resolution for atimes, since we don't even write the atime to disk more frequently than every 60s and keep an extra padding field for future use? Should we work to add more space for padding fields to mdt_rec_reint in a compatible manner to 2.3 so that the use of the *time_ns fields in 2.4 will not consume all of the available space?</p> <p>I think an initial mental exercise that would be good to perform in this case is to try and think of any fields that might need to be added in the future. If we can readily think of anything that would be handy or obvious to add (I can't, but given my inexperience with Lustre and high-performance computing in general I don't think I count), then it would imply that work should begin to add more space to mdt_rec_reint. Otherwise, if there are no extensions that may have to be made to mdt_rec_reint in the near future (as I suspect the case may be; prophecies about software are hard to tell), then I posit that at least removal of the atime_ns field is not the correct choice. While it is only updated on-disk every 60s, there is apparently a desire or need by some to have this kind of resolution for their timestamps, and at least having the nanosecond times could tell one of these highly-granular-time-needing users the most recent atime (down to the nanosecond) within that 60s between disk updates. Leaving atime out of the nanosecond club when mtime and ctime have nanoseconds also seems somewhat inconsistent.</p> <p>Since I tend favor compatability with future expansion, I would vote for working to add more space for padding fields into mdt_rec_reint in a manner compatible with 2.3. However, I don't know quite how much work that will entail; if it would be quite an undertaking, then it would seem prudent to instead leave the structures as-is and cross the bridge of expanding their size later when some fancy new feature requires it.</p> <p>Landed for 2.1.3 and 2.3</p> <p>Peter, just the patch to reserve the flag for this feature was landed, to avoid conflicts with other features being developed. The actual code is not landed yet. </p> <p>We need to re-make the nanosecond timestamp patch against the variable sized LVB patch.</p> <p>Do we want to support it in Lustre-2.5 ?</p> <p>Are any plans exist currently to rewrite the patch against the variable sized LVB patch?</p> <p>This overlaps with the 64 bit time work I have been doing. Now that we support ktime_t this can easily be handled. Just as a note DO NOT use 32 bit fields for nanoseconds. This with not work after 2038 and due to that upstream will reject the patch.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> thanks a lot for answer! I am going to look <a href="https://jira.whamcloud.com/browse/LU-9019" title="Migrate lustre to standard 64 bit time kernel API" class="issue-link" data-issue-key="LU-9019"><del>LU-9019</del></a> now.</p> <p>James, I don't understand your comment. Why would we ever want <b>more</b> than a 32-bit field for nanoseconds? Surely there can't be more than 2^32 nanoseconds in a second? I understand that there can be leap seconds and other time adjustments that might result in over 10^9 nanoseconds in a second, but using a full 64-bit field for nanoseconds in the network protocol is just a waste of space.</p> <p>Oh I see. You want to basically transmit struct timespec64 over the wire. I was thinking in terms of nanoseconds since the epoch being transmitted.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> So, do you think <a href="https://jira.whamcloud.com/browse/LU-9019" title="Migrate lustre to standard 64 bit time kernel API" class="issue-link" data-issue-key="LU-9019"><del>LU-9019</del></a> helps "transmit struct timespec64 over the wire" somehow?</p> <p>Hmm, when does ns-since-epoch overflow? Maybe 2 extra bits from the 2^30 ns in a 32-bit field... That would simplify the protocol change, and give us 4*140 years extra?</p> <p>2^32 -1 nanoseconds gives us 4.294967295 seconds until overflow.  So using just 32 bit nanoseconds time stamps are not very useful. As Atrem pointed out LNet already sends 64 bit time in seconds. We can use a 32 bit field to add nanosecond value along the already used seconds send. That is why I compared it to struct timespec64 = { time64_t tv_sec; long tv_nsec }</p> <p>Atrem all the needed infrastructure to support the linux kernel 64 bit time handling has been merged to the latest lustre. Just don't use the cfs time wrappers since they will be going away.</p> <p>Clearly I wasn't thinking of <b>only</b> a __u32 nanoseconds timestamp, since that won't even last until the end of this comment, let alone to Y2038, but rather in addition to the existing 64-bit seconds field, as was discussed in the original description.</p> <p>On the other hand, passing the entire timestamp as a 64-bit nanosecond timestamp gives us roughly 292 years before <em>signed</em> overflow, and could simplify the wire protocol change (no need to change the size of the fields) with the added complexity that there would need to be a connection flag to indicate if this client is sending timestamps in seconds or nanoseconds. Potentially we could just assume any value larger than 2^32 is going to be nanoseconds (it is unlikely that any current Lustre releases would still be running in 20 years, and conversely 2^60 ns is needed to get to 2006, so they are very unlikely to conflict), but using an <tt>OBD_CONNECT_NANOSECONDS</tt> flag is not <em>too</em> hard. The main complexity is that this flag would need to be checked in many places, possibly places where the client export is not easily available, so just making a decision based on the size of the timestamp is relatively safe, and the old seconds format could eventually be deprecated with little effort.</p> <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53313" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53313</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1158" title="nanosecond timestamp support for Lustre" class="issue-link" data-issue-key="LU-1158">LU-1158</a> general: support nanosecond timestamps<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3dc43cd08efc8347cdceed3939be7c33256a081c</p> Related LU-9019 LU-12922 LU-4050 LU-10934 LU-11971 LUDOC-92 LU-1619 Development Rank 1|hzv61z: Rank (Obsolete) 4515 Story Points 3.0