https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11695 Lustre <p>Currently, to get uptodate LSOM data, we need to set llite.*.xattr_cache=0 to disable the xattr cache on client completely. This may lead that other kind of xattr can not be cached on client too.</p> <p>This ticket mainly focues on the heavy-weight solution to disable caching only for LSOM xattr data ("trusted.som") on client.</p> LU-11695

disabling the xattr cache on client for LSOM

Improvement Major Resolved Fixed Qian Yingjin Qian Yingjin Fri, 23 Nov 2018 08:06:42 +0000 Tue, 28 Mar 2023 13:08:12 +0000 Mon, 11 Jul 2022 12:56:27 +0000 Lustre 2.16.0 Lustre 2.15.3 0 6 <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33711" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33711</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11695" title="disabling the xattr cache on client for LSOM" class="issue-link" data-issue-key="LU-11695"><del>LU-11695</del></a> som: disabling xattr cache for LSOM on client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6fe19dcc84106eddfa6616d411b58e516cbaaaf2</p> <p>As I mentioned previously, there is no need to disable the xattr cache for <tt>trusted.som</tt> completely. Virtually all files in the filesystem will not be recently modified, so disabling the cache just means more overhead for the common case where the <tt>trusted.som</tt> xattr is valid. Also, the current implementation of LSOM is mainly intended to be used from the MDS by scanning the MDT disk filesystem directly, not to be used on the client, so adding overhead on the client to manage what is supposed to be "lazy" doesn't make sense. </p> <p>Even if the <tt>trusted.som</tt> xattr is fetched repeatedly, there is still a chance that it is out-of-date (e.g. file has not been closed, <tt>llsom_sync</tt> is not running, or MDS is an older version without LSOM), so we can't ever rely on it to be completely accurate, and shouldn't try too hard to make it so. The file size will normally be accurate shortly after close, it is only the blocks count which may be inaccurate. If the application depends on the blocks count to be relatively accurate, then the userspace application should check something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct lustre_som_attrs lsom; rc = lgetxattr(file, <span class="code-quote">"trusted.som"</span>, &amp;lsom, sizeof(lsom)); size = lsom.lsa_size; blocks = lsom.lsa_blocks; <span class="code-comment">/* what unit is <span class="code-quote">"lsa_blocks"</span> in? it should be documented */</span> <span class="code-comment">/* <span class="code-keyword">if</span> xattr fetch failed or is <span class="code-object">short</span>, or <span class="code-keyword">if</span> we care about blocks being relatively accurate */</span> <span class="code-keyword">if</span> (rc &lt; sizeof(lsom) || size &gt; 1048576 &amp;&amp; lsom.lsa_blocks == 0 &amp;&amp; lsom.lsa_valid != SOM_FL_STRICT) { <span class="code-comment">/* LSOM data is likely stale (unless file is totally empty) */</span> fd = open(file, O_RDONLY); <span class="code-keyword">if</span> (fd &gt;= 0) { rc = fstat(fd, st); size = st.st_size; blocks = st.st_blocks; <span class="code-comment">/* closing the file will update the LSOM data <span class="code-keyword">for</span> next time */</span> close(fd); } } </pre> </div></div> <p>Two lighter-weight solutions exist that will still allow accurate <tt>trusted.som</tt> xattrs to be fetched to the client:</p> <ul class="alternate" type="square"> <li>cancel the mdc DLM locks via <tt>lctl set_param ldlm.namespaces.<b>mdc</b>.lru_size=clear</tt> to discard all locks. This is enough for testing that LSOM works, which is (IMHO) mostly what the "<tt>lfs getsom</tt>" interface is for. I don't think that applications will be using "<tt>lfs getsom</tt>" or the <tt>trusted.som</tt> xattr directly, and it isn't needed for the <tt>lfs find</tt> or <tt>statx()</tt> interface in the future. Note that <tt>trusted.som</tt> is only accessible to the root user, so this makes it not very useful for regular users, and a much better reason to implement <tt>statx()</tt> for regular users instead of trying to guess.</li> </ul> <ul class="alternate" type="square"> <li>implement a heuristic on the client to re-fetch the <tt>trusted.som</tt> xattr if the mtime/ctime of the inode is within the past e.g. 15 minutes (longer than what <tt>llsom_sync</tt> needs to update <tt>trusted.som</tt>, but use it from cache if the inode is older than this. The MDS will already keep the inode up-to-date for the mtime/ctime (which is needed for POSIX), and if <tt>llsom_sync</tt> hasn't update <tt>trusted.som</tt> in this time, then it is likely that it is not running and LSOM will not be more uptodate than what the client has. This will allow 99.999999% of (not-just-created) files to have the <tt>trusted.som</tt> xattr fetched on lookup and cached on the client, rather than making each access generate another RPC (which could as well get the accurate file size directly from the OSTs).</li> </ul> <p>I think at least currently, there is no requirement to use the client side interface to acess the LSOM. So, I'd suggest to delay the implement of this API until we started to see some real requirement for it. After all, LSOM is designed for MDT scanning, not client side accessment. And I think Yingjin's patch is enough to fix the test problem for now.</p> <p>Now that the statx() patch is landing, I think this patch should be abandoned, and this ticket closed. </p> <p>It turns out that the <tt>trusted.som</tt> xattr is not used directly on the client for anything except "<tt>lfs getsom</tt>", so keeping it in the xattr cache doesn't help any real workload, and only serves to confuse tests that are using <tt>getsom</tt> and see a cached value that is stale.</p> <p>I've un-abandoned patch <a href="https://review.whamcloud.com/33711" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33711</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11695" title="disabling the xattr cache on client for LSOM" class="issue-link" data-issue-key="LU-11695"><del>LU-11695</del></a> som: disabling xattr cache for LSOM on client</tt>", but it will need to be rebased to resolve conflicts.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/33711/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33711/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11695" title="disabling the xattr cache on client for LSOM" class="issue-link" data-issue-key="LU-11695"><del>LU-11695</del></a> som: disabling xattr cache for LSOM on client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 192902851d73ec246af92a2ff7be8f23b08c4343</p> <p>Landed for 2.16</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49952" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49952</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11695" title="disabling the xattr cache on client for LSOM" class="issue-link" data-issue-key="LU-11695"><del>LU-11695</del></a> som: disabling xattr cache for LSOM on client<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: e32b3e958a1210c8b082668a7afc936c3e5f16fe</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49952/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49952/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11695" title="disabling the xattr cache on client for LSOM" class="issue-link" data-issue-key="LU-11695"><del>LU-11695</del></a> som: disabling xattr cache for LSOM on client<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: dcd842e870a099e18a9afcb817f382648c96bed0</p> Related LU-15431 LU-15609 LU-15593 LU-12332 Development Rank 1|i006t3: Rank (Obsolete) 9223372036854775807