[LU-14387] sanity-quota tests fail with “lfs: failed for '/mnt/lustre': Not a directory”

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.14.0
• Severity: 3
• Assignee: WC Triage
• Reporter: James Nunez (Inactive)

Description

We have several sanity-quota tests failing with “lfs: failed for '/mnt/lustre': Not a directory” seen in the suite_log.

Looking at the suite log for https://testing.whamcloud.com/test_sets/4b810abf-1814-44f1-9e20-c2a7c022e168, we see sanity-quota test_3a, 3b and 3c all fail in the same code with

Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1353:test_3a()
 

Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3b: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1435:test_3b()

Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3c: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1551:test_3c()
 

We see the same lfs error for sanity-quota test 50, but it does not cause the test to fail

== sanity-quota test 50: Test if lfs find --projid works ============================================= 08:18:48 (1611562728)
CMD: trevis-35vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
Waiting for local destroys to complete
Creating test directory
CMD: trevis-35vm3,trevis-35vm4 lctl set_param fail_val=0 fail_loc=0
fail_val=0
fail_loc=0
fail_val=0
fail_loc=0
lfs project -sp 1 /mnt/lustre/d50.sanity-quota/dir1
lfs project -sp 2 /mnt/lustre/d50.sanity-quota/dir2
lfs: failed for '/mnt/lustre': Not a directory
Delete files...
Wait for unlink objects finished...

In the client (vm1) console log, we see the same errors for each of these tests including for sanity-quota test 50

[109788.832840] Lustre: DEBUG MARKER: Files for project (1000):
[109790.360281] LustreError: 559036:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x38af:0x0]: rc = -22
[109790.362691] LustreError: 559036:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 3 previous similar messages
[109790.364490] LustreError: 559036:0:(llite_lib.c:2837:ll_prep_inode()) new_inode -fatal: rc -22
[109790.366067] LustreError: 559036:0:(llite_lib.c:2837:ll_prep_inode()) Skipped 3 previous similar messages
[109791.258011] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT 
[109791.992042] Lustre: DEBUG MARKER: sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT

In the same test session, sanity-quota test 4a fails and has the above errors on the client, but does not have the LFS error message. We also see this error message for racer test 1, but racer passes.

History
07 AUG 2020 - We started seeing the cl_file_inode_init() error message in racer test 1 with https://testing.whamcloud.com/test_sets/2797def7-e94b-4f7c-aa18-9857a83a4b2e (racer test 1 does not fail)
07 AUG 2020 - Seeing the cl_file_inode_init() error message, along with other errors, in conf-sanity test 51 with https://testing.whamcloud.com/test_sets/41c6f606-f1ff-4809-87ed-c64d411e2414 (conf-sanity test 51 does not fail)
07 AUG 2020 - Seeing the cl_file_inode_init() error message in sanity-quota test 50 with https://testing.whamcloud.com/test_sets/7ab7eb3d-7338-4fb9-b9fd-9ee1eea5064a (sanity-quota test 50 does not fail)
08 JAN 2021 - first sanity-quota test 3b failure with cl_file_inode_init() (and lfs )error message at https://testing.whamcloud.com/test_sets/14b28d89-dde6-4c48-8125-cdb8f954ef6c
08 JAN 2021 - first sanity-quota test 3a failure with cl_file_inode_init() (and lfs )error message at https://testing.whamcloud.com/test_sets/4b810abf-1814-44f1-9e20-c2a7c022e168

Issue Links

• is related to: LU-15744 sanity-quota test_3a: ldlm_lockd.c:719:ldlm_handle_ast_error()) ### client (nid 10.240.42.19@tcp) returned error from blocking AST
• is related to: LU-14279 sanity-quota test_3b: write success, but expect EDQUOT