https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15391 Lustre <p>Suppose one has two processes, one on each of two nodes, and each open the same file and lock it via llapi_group_lock(3) using the same group, and the two processes then attempt to read and write to non-overlapping extents.</p> <p>Since they're using group lock, I understand that neither node will be aware of the writes on the other node.  I would expect that this means that the extents that each node is "assigned" for writing need to be either page-aligned or even maybe stripe-aligned, and that not choosing extents this way would result in reads possibly returning stale data.</p> <p>Is that correct?  What is the required alignment?  Are there other requirements?</p> LU-15391

lustre group lock usage and restrictions

Question/Request Minor Open Unresolved Patrick Farrell Olaf Faaland llnl Wed, 22 Dec 2021 13:22:23 +0000 Mon, 22 Aug 2022 18:04:18 +0000 0 5 <p>I've submitted this as an LU-ticket instead of LUDOC- because I would think the best place to document any requirements would be in the llapi_group_lock(3) man page.  But if that's not the best place, then please modify the ticket appropriately.</p> <p>Patrick</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Olaf,</p> <p>You're absolutely right that we need to update the man page and this is a good context in which to do it.</p> <p>You're also largely right about group locks.  The stale data issue is particularly significant - All nodes sharing a group lock have a full read/write lock on the file, so they consider any data they have to be the authoritative copy (because it's under a write lock!).</p> <p>So if any part of the file is updated one place and then read in another, or read and then updated elsewhere, stale data can result as you'd expect.  The granularity for this is <b>page</b> granularity, so if I/Os 'overlap' at the level of pages, they are subject to this stale data risk.  And if two clients do partial updates of the same page, read-modify-write means various intermixings of new and old data are possible depending on timing.</p> <p>So in general it's not safe to overlap any accesses (from different nodes - local to one node the page cache handles it and things work normally) with a write while holding a group lock.  Read-read is safe, but write-read, read-write, and even write-write cannot be safe.  (In the case of write-write, the issue is that you do not know if the first write is on disk when you start the second.  If you do page aligned writes, you can make write-write safe by using sync() calls.  But non-page-aligned writes include a read, so stale data issues apply (the read is to finish the partial page - Lustre always writes full pages to disk, so if a partial page write is attempted, it reads up the remainder of the page from disk first).)</p> <p>One other thing - while a file is open with a group lock, the file size (the size shown by 'stat' or if you're using O_APPEND) may be inaccurate if the file has been extended or truncated.  Generally it would just be potentially stale, but there are various options.  In general size cannot be trusted if you're using a group lock and writing to the file.</p> <p>One strategy I've seen used is to detect overlapping accesses in a userspace library, and when they are detected, to release the group lock on all nodes, then re-acquire it.  This syncs any dirty data to disk and clears the file from cache on all the nodes, so it's very expensive.</p> <p>In general, if you page aligned your writes and <b>only</b> wrote to the file, you'd be fine.  Not much else is really safe except read only access, and if your access is read only, why use a group lock?</p> <p>It would help me if you ask any questions or request any clarifications if this is unclear. (or even if it's confusing but you can figure it out, maybe point that out - Consider it a first review of some of the content and possible phrasing for the man page <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> ).</p> <p>Hi Patrick,</p> <p>Thanks, this is very helpful.  What you wrote makes sense to me.  I had forgotten about lockahead, so I suggested that to the person who was interested in grouplock as a safer alternative.</p> <p>-Olaf</p> <p>Hi Patrick,</p> <p>&gt; The granularity for this is page granularity</p> <p>In the case where the client and server have different page sizes, whose page size is the relevant one?</p> <p>thanks,<br/> Olaf</p> Related Development Rank 1|i02d5b: Rank (Obsolete) 9223372036854775807