[LU-15193] qsd_op_begin: more than 8 qids enforced for a transaction?

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.4
• Affects Version/s: Lustre 2.12.5
• Environment: CentOS 7.9, Lustre 2.12.7 on clients
• Severity: 3
• Assignee: Feng Lei
• Reporter: Stephane Thiell

Description

This morning, we hit the following problem (new for us) on Fir (2.12.5 servers, 2.12.7 clients):

# rmdir /scratch/users/ragoglia/csATAC/For_Rachel
rmdir: failed to remove ‘/scratch/users/ragoglia/csATAC/For_Rachel’: Invalid argument

 
More info on this directory:

[root@fir-rbh01 ~]# ls -lisa /scratch/users/ragoglia/csATAC/For_Rachel
total 8
198162765779658672 4 drwxr-xr-x  2 atrev    wjg      4096 Oct 20 02:58 .
198162765779658671 4 drwxrwxr-x+ 3 ragoglia hbfraser 4096 Sep  5  2019 ..

[root@fir-rbh01 ~]# lfs getdirstripe /scratch/users/ragoglia/csATAC/For_Rachel
lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none

[root@fir-rbh01 ~]# lfs project -d /scratch/users/ragoglia/csATAC/For_Rachel
259557 P /scratch/users/ragoglia/csATAC/For_Rachel

[root@fir-rbh01 ~]# lfs getdirstripe /scratch/users/ragoglia/csATAC
lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none

[root@fir-rbh01 ~]# rmdir /scratch/users/ragoglia/csATAC/For_Rachel
rmdir: failed to remove ‘/scratch/users/ragoglia/csATAC/For_Rachel’: Invalid argument

MDS logs show:

Nov 03 09:52:49 fir-md1-s3 kernel: LustreError: 103307:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction?
Nov 03 10:10:09 fir-md1-s3 kernel: LustreError: 103781:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction?
Nov 03 10:10:35 fir-md1-s3 kernel: LustreError: 103781:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction?
Nov 03 10:10:52 fir-md1-s3 kernel: LustreError: 103697:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction?
Nov 03 10:15:55 fir-md1-s3 kernel: LustreError: 103721:0:(qsd_handler.c:884:qsd_op_begin()) fir-MDT0002: more than 8 qids enforced for a transaction?

Which looks like LU-12388
 
My colleague found a workaround for rmdir: set project ID 0 before rmdir, and it worked:

# lfs project -d  /scratch/users/ragoglia/csATAC/For_Rachel/
259557 P /scratch/users/ragoglia/csATAC/For_Rachel/
# lfs project -p 0  /scratch/users/ragoglia/csATAC/For_Rachel/
# lfs project -d  /scratch/users/ragoglia/csATAC/For_Rachel/
    0 P /scratch/users/ragoglia/csATAC/For_Rachel/
# rmdir /scratch/users/ragoglia/csATAC/For_Rachel/
#

I believe this is the only occurrence we've seen of this issue so far. Let me know if additional logs would be helpful the next time we hit this.
 

Issue Links

• is related to: LU-12388 expand QUOTA_MAX_TRANSIDS for Project Quota?

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49611/
Subject: LU-15193 quota: expand QUOTA_MAX_TRANSIDS to 12
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: c20d23cd92c5bc748a618e9ed96e6eddd794ab45

[Stephane Thiell]

Sounds good, makes sense, I will remember for next time, thanks Andreas and Feng!

[Gerrit Updater]

"Feng Lei <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49611
Subject: LU-15193 quota: expand QUOTA_MAX_TRANSIDS to 12
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 44db441ca5ce7395cd059e3230b4ae684db01830

[Andreas Dilger]

It would be nice to have this patch backported to 2.15 LTS. Thanks!

For simple patches like this you can usually cherry-pick the patch directly to b2_15 within Gerrit. Click the "[Cherry-pick]" button on the patch, add "b2_15" for the branch name, edit the commit message to add labels "Lustre-change:" (from "Reviewed-on:") and "Lustre-commit:" (from the "cherry-picked" line at the end), and remove the "Tested-by:" and "Reviewed-by: Oleg Drokin" lines.

As an added benefit, if I don't do the cherry-pick the patch myself, I'm able to review it and it can be landed more quickly instead of waiting for someone else to review it.

[Stephane Thiell]

It would be nice to have this patch backported to 2.15 LTS. Thanks!