[LU-2548] After upgrade from 1.8.8 to 2.4 hit qmt_entry.c:281:qmt_glb_write()) $$$ failed to update global index, rc:-5

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0
• Labels: HB
• Environment: before upgrade: client and server are running 1.8.8; after upgrade: client and server are running lustre-master build#1141
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Sarah Liu

Description

After clean upgrade server and client from 1.8.8 to 2.4, I enabled quota with following steps:
1. before setup Lustre: tunefs.lustre --quota mdsdev/ostdev
2. after setup Lustre: lctl conf_param lustre.quota.mdt=ug
lctl conf_param lustre.quota.ost=ug

then do iozone got this error:

upgrade-downgrade : @@@@@@ FAIL: iozone did not fail with EDQUOT
{noforamt}

found errors in mds dmesg:

Lustre: DEBUG MARKER: ===== Pass ==================================================================
Lustre: DEBUG MARKER: ===== Check Lustre quotas usage/limits ======================================
Lustre: DEBUG MARKER: ===== Verify the data =======================================================
Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt
LDISKFS-fs warning (device sdb1): ldiskfs_block_to_path: block 1852143205 > max in inode 24537
LustreError: 7867:0:(qmt_entry.c:281:qmt_glb_write()) $$$ failed to update global index, rc:-5 qmt:lustre-QMT0000 pool:0-md id:60001 enforced:1 hard:5120 soft:0 granted:1024 time:0 qunit:1024 edquot:0 may_rel:0 revoke:4297684387
LustreError: 10848:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x1 qsd:lustre-MDT0000 qtype:usr id:60001 enforced:1 granted:3 pending:0 waiting:2 req:1 usage:3 qunit:0 qtune:0 edquot:0
Lustre: DEBUG MARKER: upgrade-downgrade : @@@@@@ FAIL: iozone did not fail with EDQUOT
LDISKFS-fs warning (device sdb1): ldiskfs_block_to_path:
LDISKFS-fs warning (device sdb1): ldiskfs_block_to_path: block 1852143205 > max in inode 24537
LustreError: 10877:0:(qmt_entry.c:281:qmt_glb_write()) $$$ failed to update global index, rc:-5 qmt:lustre-QMT0000 pool:0-md id:60001 enforced:1 hard:5120 soft:0 granted:1026 time:0 qunit:1024 edquot:0 may_rel:0 revoke:4297684387
LustreError: 7577:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x2 qsd:lustre-MDT0000 qtype:usr id:60001 enforced:1 granted:3 pending:0 waiting:0 req:1 usage:2 qunit:1024 qtune:512 edquot:0
LDISKFS-fs warning (device sdb1): ldiskfs_block_to_path: block 1852143205 > max in inode 24537
LDISKFS-fs warning (device sdb1): ldiskfs_block_to_path: block 1852143205 > max in inode 24537
block 1768711539 > max in inode 24538

Issue Links

• is related to: LU-2688 add quota upgrade checks to conf_sanity.sh test_32

Comments

[Sarah Liu]

Niu, I tried upgrading 1.8->2.4 again and it can be reproduced.

[Sarah Liu]

Niu, this time I upgraded to the latest tag-2.3.58, that's a different build from the first time.

I will keep you updated when I finish upgrading from 2.1 to 2.4 and try again 1.8 to 2.4 to see if it happens every time.

[Niu Yawei (Inactive)]

This time the error messages are different from the first time. Sarah, is it reproduceable on 2.1 to 2.4 upgrading? How often did it happen on 1.8 -> 2.4 upgrading? Thanks.

[Sarah Liu]

debug and dmesg logs from MDS

[Sarah Liu]

will keep you updated.

[Niu Yawei (Inactive)]

Hi, Sarah. Could you enable TRACE to collect a debug log for this failure? Thanks.