[LU-14387] sanity-quota tests fail with “lfs: failed for '/mnt/lustre': Not a directory” Created: 29/Jan/21  Updated: 03/Aug/22

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.14.0
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: James Nunez (Inactive) Assignee: WC Triage
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Related
is related to LU-14279 sanity-quota test_3b: write success, ... Resolved
is related to LU-15744 sanity-quota test_3a: ldlm_lockd.c:71... Open
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

We have several sanity-quota tests failing with “lfs: failed for '/mnt/lustre': Not a directory” seen in the suite_log.

Looking at the suite log for https://testing.whamcloud.com/test_sets/4b810abf-1814-44f1-9e20-c2a7c022e168, we see sanity-quota test_3a, 3b and 3c all fail in the same code with

Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1353:test_3a()
 
Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3b: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1435:test_3b()
Files for project (1000):
lfs: failed for '/mnt/lustre': Not a directory
 sanity-quota test_3c: @@@@@@ FAIL: write success, but expect EDQUOT 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6273:error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:159:quota_error()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1297:test_block_soft()
  = /usr/lib64/lustre/tests/sanity-quota.sh:1551:test_3c()
 

We see the same lfs error for sanity-quota test 50, but it does not cause the test to fail

== sanity-quota test 50: Test if lfs find --projid works ============================================= 08:18:48 (1611562728)
CMD: trevis-35vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
Waiting for local destroys to complete
Creating test directory
CMD: trevis-35vm3,trevis-35vm4 lctl set_param fail_val=0 fail_loc=0
fail_val=0
fail_loc=0
fail_val=0
fail_loc=0
lfs project -sp 1 /mnt/lustre/d50.sanity-quota/dir1
lfs project -sp 2 /mnt/lustre/d50.sanity-quota/dir2
lfs: failed for '/mnt/lustre': Not a directory
Delete files...
Wait for unlink objects finished...

In the client (vm1) console log, we see the same errors for each of these tests including for sanity-quota test 50

[109788.832840] Lustre: DEBUG MARKER: Files for project (1000):
[109790.360281] LustreError: 559036:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x38af:0x0]: rc = -22
[109790.362691] LustreError: 559036:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 3 previous similar messages
[109790.364490] LustreError: 559036:0:(llite_lib.c:2837:ll_prep_inode()) new_inode -fatal: rc -22
[109790.366067] LustreError: 559036:0:(llite_lib.c:2837:ll_prep_inode()) Skipped 3 previous similar messages
[109791.258011] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT 
[109791.992042] Lustre: DEBUG MARKER: sanity-quota test_3a: @@@@@@ FAIL: write success, but expect EDQUOT

In the same test session, sanity-quota test 4a fails and has the above errors on the client, but does not have the LFS error message. We also see this error message for racer test 1, but racer passes.

History
07 AUG 2020 - We started seeing the cl_file_inode_init() error message in racer test 1 with https://testing.whamcloud.com/test_sets/2797def7-e94b-4f7c-aa18-9857a83a4b2e (racer test 1 does not fail)
07 AUG 2020 - Seeing the cl_file_inode_init() error message, along with other errors, in conf-sanity test 51 with https://testing.whamcloud.com/test_sets/41c6f606-f1ff-4809-87ed-c64d411e2414 (conf-sanity test 51 does not fail)
07 AUG 2020 - Seeing the cl_file_inode_init() error message in sanity-quota test 50 with https://testing.whamcloud.com/test_sets/7ab7eb3d-7338-4fb9-b9fd-9ee1eea5064a (sanity-quota test 50 does not fail)
08 JAN 2021 - first sanity-quota test 3b failure with cl_file_inode_init() (and lfs )error message at https://testing.whamcloud.com/test_sets/14b28d89-dde6-4c48-8125-cdb8f954ef6c
08 JAN 2021 - first sanity-quota test 3a failure with cl_file_inode_init() (and lfs )error message at https://testing.whamcloud.com/test_sets/4b810abf-1814-44f1-9e20-c2a7c022e168


Generated at Sat Feb 10 03:09:17 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.