https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15193 Lustre <p>This morning, we hit the following problem (new for us) on Fir (2.12.5 servers, 2.12.7 clients):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rmdir /scratch/users/ragoglia/csATAC/For_Rachel rmdir: failed to remove ‘/scratch/users/ragoglia/csATAC/For_Rachel’: Invalid argument </pre> </div></div> <p> <br/> More info on this directory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 ~]# ls -lisa /scratch/users/ragoglia/csATAC/For_Rachel total 8 198162765779658672 4 drwxr-xr-x 2 atrev wjg 4096 Oct 20 02:58 . 198162765779658671 4 drwxrwxr-x+ 3 ragoglia hbfraser 4096 Sep 5 2019 .. [root@fir-rbh01 ~]# lfs getdirstripe /scratch/users/ragoglia/csATAC/For_Rachel lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none [root@fir-rbh01 ~]# lfs project -d /scratch/users/ragoglia/csATAC/For_Rachel 259557 P /scratch/users/ragoglia/csATAC/For_Rachel [root@fir-rbh01 ~]# lfs getdirstripe /scratch/users/ragoglia/csATAC lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none [root@fir-rbh01 ~]# rmdir /scratch/users/ragoglia/csATAC/For_Rachel rmdir: failed to remove ‘/scratch/users/ragoglia/csATAC/For_Rachel’: Invalid argument </pre> </div></div> <p>MDS logs show:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 03 09:52:49 fir-md1-s3 kernel: LustreError: 103307:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? Nov 03 10:10:09 fir-md1-s3 kernel: LustreError: 103781:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? Nov 03 10:10:35 fir-md1-s3 kernel: LustreError: 103781:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? Nov 03 10:10:52 fir-md1-s3 kernel: LustreError: 103697:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? Nov 03 10:15:55 fir-md1-s3 kernel: LustreError: 103721:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? </pre> </div></div> <p>Which looks like <a href="https://jira.whamcloud.com/browse/LU-12388" title="expand QUOTA_MAX_TRANSIDS for Project Quota?" class="issue-link" data-issue-key="LU-12388"><del>LU-12388</del></a><br/>  <br/> My colleague found a workaround for rmdir: set project ID 0 before rmdir, and it worked:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs project -d /scratch/users/ragoglia/csATAC/For_Rachel/ 259557 P /scratch/users/ragoglia/csATAC/For_Rachel/ # lfs project -p 0 /scratch/users/ragoglia/csATAC/For_Rachel/ # lfs project -d /scratch/users/ragoglia/csATAC/For_Rachel/ 0 P /scratch/users/ragoglia/csATAC/For_Rachel/ # rmdir /scratch/users/ragoglia/csATAC/For_Rachel/ # </pre> </div></div> <p>I believe this is the only occurrence we've seen of this issue so far. Let me know if additional logs would be helpful the next time we hit this.<br/>  </p> CentOS 7.9, Lustre 2.12.7 on clients LU-15193

qsd_op_begin: more than 8 qids enforced for a transaction?

Bug Minor Resolved Fixed Feng Lei Stephane Thiell Wed, 3 Nov 2021 18:33:22 +0000 Mon, 11 Dec 2023 07:19:21 +0000 Sat, 11 Jun 2022 15:31:20 +0000 Lustre 2.12.5 Lustre 2.16.0 Lustre 2.15.4 0 7 <p>It looks like a trivial fix. It would also be useful if the <tt>qsd_op_begin()</tt> error message was updated to also print the FID, so that this is easier to debug in the future. </p> <p>It may not be <em>quite</em> as trivial as I thought. The trivial solution is to increase the max quotas to 12 - ( &lt;old,new&gt; or &lt;parent,child&gt; x &lt;user,group,project&gt; x &lt;block,inode&gt;, but this may needlessly increase the transaction credits and hurt performance. </p> <p>In the case of "rmdir" it isn't clear why the quota of the parent directory would be updated (it only removes a name from the leaf block, and updates the link count on the inode, so until we have directory shrink there would be no changes to the parent ID quotas), so (IMHO) there should only be at most 6 different IDs involved in this transaction. </p> <p>Simple reproducer below.</p> <p>It looks like the owner/group of the parent directory has to be different than the owner/group of the directory to be removed, and to trigger the problem, the project IDs have to be different too.</p> <p>From a directory that has project ID set by default (and inherited):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 LU-15193]# lfs project -d . 282232 P . [root@fir-rbh01 LU-15193]# mkdir parent [root@fir-rbh01 LU-15193]# chown atrev.wjg parent [root@fir-rbh01 LU-15193]# mkdir parent/dir [root@fir-rbh01 LU-15193]# chown sthiell.ruthm parent/dir [root@fir-rbh01 LU-15193]# ls -lisa parent total 12 180150013517627412 4 drwxr-xr-x 3 atrev wjg 4096 Nov 3 13:50 . 180150013517627393 4 drwxr-xr-x 3 root root 4096 Nov 3 13:49 .. 180150013517627413 4 drwxr-xr-x 2 sthiell ruthm 4096 Nov 3 13:50 dir [root@fir-rbh01 LU-15193]# lfs project -p 215845 parent/dir [root@fir-rbh01 LU-15193]# lfs project -d parent parent/dir 282232 P parent 215845 P parent/dir [root@fir-rbh01 LU-15193]# rmdir parent/dir rmdir: failed to remove 'parent/dir': Invalid argument </pre> </div></div> <p>In practice, for us, that's definitely a very rare case, that's probably why we haven't seen it before.</p> <p>"Feng, Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45456" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45456</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15193" title="qsd_op_begin: more than 8 qids enforced for a transaction?" class="issue-link" data-issue-key="LU-15193"><del>LU-15193</del></a> quota: expand QUOTA_MAX_TRANSIDS to 12<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c995e003c75b15452d48fc94a31165eb194d4455</p> <p>Hello! We are still hitting this problem. Example from this morning using a client running 2.12.8 and servers running 2.12.7:</p> <p>Client-side:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sh02-hn01 VASP]# pwd /scratch/users/dasc/Projects/BATT-RIXS/VASP [root@sh02-hn01 VASP]# ls -al total 12 drwxrwxr-x+ 3 311749 tpd 4096 May 27 2021 . drwxrwxr-x+ 4 311749 tpd 4096 Jun 16 2020 .. drwxr-xr-x 2 292018 32269 4096 May 27 2021 Na2Mn3O7 [root@sh02-hn01 VASP]# rm -Rf Na2Mn3O7/ [root@sh02-hn01 VASP]# ls -al total 12 drwxrwxr-x+ 3 311749 tpd 4096 May 27 2021 . drwxrwxr-x+ 4 311749 tpd 4096 Jun 16 2020 .. drwxr-xr-x 2 292018 32269 4096 May 27 2021 Na2Mn3O7 [root@sh02-hn01 VASP]# ls -al Na2Mn3O7/ total 8 drwxr-xr-x 2 292018 32269 4096 May 27 2021 . drwxrwxr-x+ 3 311749 tpd 4096 May 27 2021 .. [root@sh02-hn01 VASP]# rmdir Na2Mn3O7/ rmdir: failed to remove 'Na2Mn3O7/': Invalid argument </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs getdirstripe /scratch/users/dasc/Projects/BATT-RIXS lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none # lfs getdirstripe /scratch/users/dasc/Projects/BATT-RIXS/VASP lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none </pre> </div></div> <p>Server-side:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fir-md1-s3: Jan 27 11:04:36 fir-md1-s3 kernel: LustreError: 24993:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction? </pre> </div></div> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45456/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45456/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15193" title="qsd_op_begin: more than 8 qids enforced for a transaction?" class="issue-link" data-issue-key="LU-15193"><del>LU-15193</del></a> quota: expand QUOTA_MAX_TRANSIDS to 12<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 61481796ac85e9ab2469b8d2f4cc75088c65d298</p> <p>Landed for 2.16</p> <p>It would be nice to have this patch backported to 2.15 LTS. Thanks!</p> <blockquote> <p>It would be nice to have this patch backported to 2.15 LTS. Thanks!</p></blockquote> <p>For simple patches like this you can usually cherry-pick the patch directly to b2_15 within Gerrit. Click the "<tt><span class="error">&#91;Cherry-pick&#93;</span></tt>" button on the patch, add "<tt>b2_15</tt>" for the branch name, edit the commit message to add labels "<tt>Lustre-change:</tt>" (from "<tt>Reviewed-on:</tt>") and "<tt>Lustre-commit:</tt>" (from the "<tt>cherry-picked</tt>" line at the end), and remove the "<tt>Tested-by:</tt>" and "<tt>Reviewed-by: Oleg Drokin</tt>" lines.</p> <p>As an added benefit, if I don't do the cherry-pick the patch myself, I'm able to review it and it can be landed more quickly instead of waiting for someone else to review it.</p> <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49611" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49611</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15193" title="qsd_op_begin: more than 8 qids enforced for a transaction?" class="issue-link" data-issue-key="LU-15193"><del>LU-15193</del></a> quota: expand QUOTA_MAX_TRANSIDS to 12<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 44db441ca5ce7395cd059e3230b4ae684db01830</p> <p>Sounds good, makes sense, I will remember for next time, thanks Andreas and Feng!</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49611/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49611/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15193" title="qsd_op_begin: more than 8 qids enforced for a transaction?" class="issue-link" data-issue-key="LU-15193"><del>LU-15193</del></a> quota: expand QUOTA_MAX_TRANSIDS to 12<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: c20d23cd92c5bc748a618e9ed96e6eddd794ab45</p> Related LU-12388 Development Rank 1|i02967: Rank (Obsolete) 9223372036854775807 Severity