[LU-10753] sanity test 300c fails with 'create 5k files failed'

Duplicate — Thu, 4 Mar 2021 14:59:08 +0000

sanity test_300c fails to create 5,000 files for Lustre file systems with DNE configured and ZFS server targets. Actually, it can’t even create one file. The client test log has the following output

== sanity test 300c: chown && check ls under striped directory ======================================= 07:16:38 (1508915798)
CMD: trevis-49vm7 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
running as uid/gid/euid/egid 500/500/500/500, groups:
 [createmany] [-o] [/mnt/lustre/d300c.sanity/striped_dir/f] [5000]
open(/mnt/lustre/d300c.sanity/striped_dir/f0) error: Permission denied
total: 0 open/close in 0.01 seconds: 0.00 ops/second
 sanity test_300c: @@@@@@ FAIL: create 5k files failed

There’s nothing interesting in the console and dmesg logs on any of the nodes.

This test started failing with the ‘create 5k files failed’ error on 2017-09-29 for master tag 2.10.53. We’ve also seen this fail once for full test session DNE/ZFS testing on 2.10.1.

Logs for test sessions with this failure are at:
https://testing.hpdd.intel.com/test_sets/7ce33f5a-ba7b-11e7-9abd-52540065bddc
https://testing.hpdd.intel.com/test_sets/336e7196-12b9-11e8-a10a-52540065bddc
https://testing.hpdd.intel.com/test_sets/d61ba008-167e-11e8-bd00-52540065bddc

We see the other sanity 300* tests fail in similar ways and can open separate tickets if we think the following have distinct root cause for the failures.

sanity test 300d fails with

== sanity test 300d: check default stripe under striped directory ==================================== 06:31:06 (1516775466)
CMD: trevis-13vm4 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
open(/mnt/lustre/d300d.sanity/striped_dir/f1) error: Permission denied
total: 1 open/close in 0.01 seconds: 189.21 ops/second
 sanity test_300d: @@@@@@ FAIL: create 10 files failed

with logs at
https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc

sanity test 300e fails with

== sanity test 300e: check rename under striped directory ============================================ 01:22:13 (1519521733)
CMD: trevis-66vm8 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/a': Permission denied
touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/c': Permission denied
mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_a': Permission denied
mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_c': Permission denied
rename returned -1: No such file or directory
 sanity test_300e: @@@@@@ FAIL: rename dir under striped dir fails

with logs at:
https://testing.hpdd.intel.com/test_sets/cceb366a-19df-11e8-a6ad-52540065bddc
https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc
https://testing.hpdd.intel.com/test_sets/4b90ec14-ffb7-11e7-bd00-52540065bddc

Whamcloud Community JIRA

[LU-10753] sanity test 300c fails with 'create 5k files failed'