https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16205 Lustre <p>Two cases to support:</p> <ul class="alternate" type="square"> <li>fid2path with the encryption key</li> <li>fid2path without the encryption key</li> </ul> LU-16205

fid2path for encrypted files

Bug Minor Resolved Fixed Sebastien Buisson Sebastien Buisson encryption security Tue, 4 Oct 2022 15:49:14 +0000 Tue, 25 Apr 2023 03:42:42 +0000 Mon, 13 Mar 2023 18:59:52 +0000 Lustre 2.16.0 0 2 <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48930" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48930</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16205" title="fid2path for encrypted files" class="issue-link" data-issue-key="LU-16205"><del>LU-16205</del></a> sec: fid2path for encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dc41113a309d3eaafb7389cf9c8e31570f78ee73</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49028" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49028</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16205" title="fid2path for encrypted files" class="issue-link" data-issue-key="LU-16205"><del>LU-16205</del></a> sec: reserve flag for fid2path for encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d67702bd2f3fb55fa7099e6890ae507d06cbf0a9</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49028/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49028/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16205" title="fid2path for encrypted files" class="issue-link" data-issue-key="LU-16205"><del>LU-16205</del></a> sec: reserve flag for fid2path for encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6f74bb60ff6c58f4a2647556124c501100330f4c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48930/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48930/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16205" title="fid2path for encrypted files" class="issue-link" data-issue-key="LU-16205"><del>LU-16205</del></a> sec: fid2path for encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fa9da556ad22b1485c53cf0337dc6872d89aedfa</p> <p>Landed for 2.16</p> <p>It looks like this patch has somehow resulted in a leak of job_stats on the server that result in a message being printed at unmount time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1095.059674] Lustre: Failing over lustre-MDT0000 [ 1095.090215] LustreError: 31568:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 1095.093184] LustreError: 31568:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 20 previous similar messages [ 1095.146954] Lustre: server umount lustre-MDT0000 complete </pre> </div></div> <p>According to a <a href="https://50.222.100.39/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:&#39;2023-01-13T12:00:00.000Z&#39;,to:&#39;2023-02-11T00:00:00.000Z&#39;))&amp;_a=(columns:!(sub_test_name,test_session_job),filters:!(),index:ffc3bad0-3e61-11ec-b0b9-8ba7a9c0d3d3,interval:auto,query:(language:kuery,query:&#39;job_stat_exit%20and%20test_set_name:sanity-sec%20and%20sub_test_name:test_63&#39;),sort:!(!(sub_test_date,desc)))" class="external-link" target="_blank" rel="nofollow noopener">Kibana log search</a>, this message first appeared on 2023-01-13 for patch commit a747abbea881 (v4 of "<a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/48930/4" class="external-link" target="_blank" rel="nofollow noopener"><tt>LU-16205 sec: fid2path for encrypted files</tt></a>") and commit e53c0eaa56fe (v3 of "<a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/49153/3" class="external-link" target="_blank" rel="nofollow noopener"><tt>LU-16310 sec: Lustre/HSM on enc file with enc key</tt></a>") that was based on top of it. This repeated for each refresh of those patches, and then picked up significantly on 2023-02-03 after the patches landed.</p> <p>I picked sanity-sec test_63 to speed up the search for large date ranges, but this message is present in many other subtests on master.</p> <p>I will have a look. At first sight it looks weird that v4 of patch #48930 introduced this behavior, as compared to v3 it only modifies sanity-sec.sh, and for cosmetic changes. But a rebase was also made when pushing v4, so it might be a collateral damage.</p> <p>I pushed debug patch <a href="https://review.whamcloud.com/50275" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50275</a> to investigate this issue. It is a revert of:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-16494 fileset: check fileset for operations by fid LU-16310 sec: Lustre/HSM on enc file with enc key LU-16205 sec: fid2path for encrypted files </pre> </div></div> <p>With that build, I still get the messages about job stats when stopping an MDT target:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1329.906800] Lustre: Failing over lustre-MDT0000 [ 1329.977350] LustreError: 34261:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 1329.979183] LustreError: 34261:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 9 previous similar messages [ 1330.065565] Lustre: server umount lustre-MDT0000 complete </pre> </div></div> <p>To reproduce, I just configured job stats on a client with <tt># lctl set_param jobid_var=procname_uid</tt>, then did some IOs, then dumped job stats on the MDT with <tt>lctl get_param mdt.*.job_stats</tt>, and finally unmounted the MDT target.</p> <p>So I doubt those messages are due to the fid2path patches. I think this ticket can be re-closed.</p> <p>Ah, it seems I was mislead by my random selection of sanity-sec test_63 to find the first occurrence. Since that subtest was added in this patch it is (in hindsight) obvious that there would be no cases of the error message before the patch. I was tricked by the fact that the errors were seen before the patch landed, and then became common after it landed. I will need to re-do the searches with a different subtest that existed in the tree earlier. </p> <p>Filed <a href="https://jira.whamcloud.com/browse/LU-16639" title="job_stat_exit() should not have any items" class="issue-link" data-issue-key="LU-16639"><del>LU-16639</del></a> to track this jobid issue further. </p> Related LU-16639 Development Epic Link EX-5772 Rank 1|i031yv: Rank (Obsolete) 9223372036854775807 Severity