[LU-8340] sanity-sec test_25: /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED!

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.9.0
• Environment: Full - EL7.2 Server/EL7.2 Client; master, build# 3399
• Severity: 3
• Assignee: Kit Westneat (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/12ba69e2-3a6a-11e6-a0ce-5254006e85c2.

The sub-test test_25 failed with the following error:

/usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED!

test log:

Started clients trevis-48vm1.trevis.hpdd.intel.com,trevis-48vm2: 
CMD: trevis-48vm1.trevis.hpdd.intel.com,trevis-48vm2 mount | grep /mnt/lustre' '
10.9.6.52@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr)
10.9.6.52@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr)
CMD: trevis-48vm2 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck\" \"all -lnet -lnd -pinger\" 4 
CMD: trevis-48vm7 lctl get_param -n timeout
Using TIMEOUT=20
CMD: trevis-48vm7 lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
CMD: trevis-48vm1.trevis.hpdd.intel.com lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
enable quota as required
CMD: trevis-48vm7 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled
CMD: trevis-48vm8 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0000.quota_slave.enabled
[HOST:trevis-48vm1.trevis.hpdd.intel.com] [old_mdt_qtype:ug] [old_ost_qtype:ug] [new_qtype:ug3]
CMD: trevis-48vm7 /usr/sbin/lctl conf_param lustre.quota.mdt=ug3
CMD: trevis-48vm7 /usr/sbin/lctl conf_param lustre.quota.ost=ug3
Total disk size: 13760516  block-softlimit: 13761540 block-hardlimit: 14449617 inode-softlimit: 916168 inode-hardlimit: 961976
Setting up quota on trevis-48vm1.trevis.hpdd.intel.com:/mnt/lustre for quota_usr...
+ /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre
setquota failed: Operation not permitted
 sanity-sec test_25: @@@@@@ FAIL: /usr/bin/lfs setquota -u quota_usr -b 13761540 -B 14449617 -i 916168 -I 961976 /mnt/lustre FAILED! 

This issue was first seen on 06/19/2016 and since then it has occurred around 22 times till now (i.e. in 10 days, 22 times).

Issue Links

• is related to: LU-3291 IU UID/GID Mapping Feature

Comments

[Gerrit Updater]

Kit Westneat ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22750
Subject: LU-8340 tests: in sanity-sec 25 client nodemap must be admin
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 466a4add4b2bf92012920b0acff46745899547fd

[Peter Jones]

Kit

The debug patch has landed for over a week now. Have you been able to gather any useful info as a result?

Peter

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/21907/
Subject: LU-8340 nodemap: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 193987fbbb7595d29cec47f03f009f56caa90924

[Kit Westneat (Inactive)]

Hi Peter,

I haven't been able to reproduce the bug yet, I'll try some more this week.

Thanks,
Kit

[Peter Jones]

Kit

Has the debug patch provided the information you hoped for?

Peter

[Saurabh Tandan (Inactive)]

Another instance: Interop EL7.2 Server/2.8.0 EL7.2 Client
Server: master, build# 3418
Client: b2_8, build# 12
https://testing.hpdd.intel.com/test_sets/77f2fecc-5f00-11e6-906c-5254006e85c2

[Gerrit Updater]

Kit Westneat ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21907
Subject: LU-8340 nodemap: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 07e836a33a3cd5613fcda80944cb41bad20cecec

[Kit Westneat (Inactive)]

That's a good idea, I'll upload a debug patch as well with test params to see if I can reliably trigger it on maloo.

[Peter Jones]

Kit

Thanks for the update. Those kind of issues can be infuriating. Do you know how to leverage test parameters to easily run the test multiple times? Adding additional debug to the tests and doing that is usually how engineers manage to get to the bottom of such issues.

Peter

[Kit Westneat (Inactive)]

Hi Peter,

I've taken a look at it, but I'm not sure why it fails sometimes and not others. My gut is that it's just a test issue, but I will spend some time trying to reproduce it on my system and get a clearer answer. I don't think it's a blocker, but it would be nice to have it sorted out before the next release.

[Peter Jones]

Any update Kit? Do you think that this is something we ought to fix for 2.9 or could it wait for a future release?