[LU-8340] sanity-sec test_25: /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED!

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.9.0
• Environment: Full - EL7.2 Server/EL7.2 Client; master, build# 3399
• Severity: 3
• Assignee: Kit Westneat (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/12ba69e2-3a6a-11e6-a0ce-5254006e85c2.

The sub-test test_25 failed with the following error:

/usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED!

test log:

Started clients trevis-48vm1.trevis.hpdd.intel.com,trevis-48vm2: 
CMD: trevis-48vm1.trevis.hpdd.intel.com,trevis-48vm2 mount | grep /mnt/lustre' '
10.9.6.52@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr)
10.9.6.52@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr)
CMD: trevis-48vm2 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck\" \"all -lnet -lnd -pinger\" 4 
CMD: trevis-48vm7 lctl get_param -n timeout
Using TIMEOUT=20
CMD: trevis-48vm7 lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
CMD: trevis-48vm1.trevis.hpdd.intel.com lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
enable quota as required
CMD: trevis-48vm7 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled
CMD: trevis-48vm8 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0000.quota_slave.enabled
[HOST:trevis-48vm1.trevis.hpdd.intel.com] [old_mdt_qtype:ug] [old_ost_qtype:ug] [new_qtype:ug3]
CMD: trevis-48vm7 /usr/sbin/lctl conf_param lustre.quota.mdt=ug3
CMD: trevis-48vm7 /usr/sbin/lctl conf_param lustre.quota.ost=ug3
Total disk size: 13760516  block-softlimit: 13761540 block-hardlimit: 14449617 inode-softlimit: 916168 inode-hardlimit: 961976
Setting up quota on trevis-48vm1.trevis.hpdd.intel.com:/mnt/lustre for quota_usr...
+ /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre
setquota failed: Operation not permitted
 sanity-sec test_25: @@@@@@ FAIL: /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED! 

This issue was first seen on 06/19/2016 and since then it has occurred around 22 times till now (i.e. in 10 days, 22 times).

Issue Links

• is related to: LU-3291 IU UID/GID Mapping Feature

Comments

[Peter Jones]

Landed for 2.9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/22750/
Subject: LU-8340 tests: in sanity-sec 25 client nodemap must be admin
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1a92e40bd05c05820c902ce4ee738212a308d9e2

[Jian Yu]

Is it possible to confirm that ENABLE_QUOTA is disabled on the normal tests, but enabled on the full tests?

Yes, Kit.

For full test group, ENABLE_QUOTA=yes is specified. For review-* test groups, ENABLE_QUOTA is disabled.

[Kit Westneat (Inactive)]

Hi Peter,

I think it's an issue with the ENABLE_QUOTA flag being set. When it is set, restarting Lustre causes the client to issue quota commands, but quota permissions were disabled by nodemap at that spot. This patch reenables the admin permissions on the client, so that should fix it.

Is it possible to confirm that ENABLE_QUOTA is disabled on the normal tests, but enabled on the full tests?

Thanks,
Kit

[Gerrit Updater]

Kit Westneat ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22750
Subject: LU-8340 tests: in sanity-sec 25 client nodemap must be admin
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 466a4add4b2bf92012920b0acff46745899547fd

[Peter Jones]

Kit

The debug patch has landed for over a week now. Have you been able to gather any useful info as a result?

Peter

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/21907/
Subject: LU-8340 nodemap: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 193987fbbb7595d29cec47f03f009f56caa90924

[Kit Westneat (Inactive)]

Hi Peter,

I haven't been able to reproduce the bug yet, I'll try some more this week.

Thanks,
Kit

[Peter Jones]

Kit

Has the debug patch provided the information you hoped for?

Peter

[Saurabh Tandan (Inactive)]

Another instance: Interop EL7.2 Server/2.8.0 EL7.2 Client
Server: master, build# 3418
Client: b2_8, build# 12
https://testing.hpdd.intel.com/test_sets/77f2fecc-5f00-11e6-906c-5254006e85c2