[LU-10753] sanity test 300c fails with 'create 5k files failed'

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.7
• Affects Version/s: Lustre 2.10.1, Lustre 2.11.0, Lustre 2.12.0, Lustre 2.13.0, Lustre 2.14.0
• Labels: dne zfs
• Severity: 3
• Assignee: Lai Siyao
• Reporter: James Nunez (Inactive)

Description

sanity test_300c fails to create 5,000 files for Lustre file systems with DNE configured and ZFS server targets. Actually, it can’t even create one file. The client test log has the following output

== sanity test 300c: chown && check ls under striped directory ======================================= 07:16:38 (1508915798)
CMD: trevis-49vm7 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
running as uid/gid/euid/egid 500/500/500/500, groups:
 [createmany] [-o] [/mnt/lustre/d300c.sanity/striped_dir/f] [5000]
open(/mnt/lustre/d300c.sanity/striped_dir/f0) error: Permission denied
total: 0 open/close in 0.01 seconds: 0.00 ops/second
 sanity test_300c: @@@@@@ FAIL: create 5k files failed

There’s nothing interesting in the console and dmesg logs on any of the nodes.

This test started failing with the ‘create 5k files failed’ error on 2017-09-29 for master tag 2.10.53. We’ve also seen this fail once for full test session DNE/ZFS testing on 2.10.1.

Logs for test sessions with this failure are at:
https://testing.hpdd.intel.com/test_sets/7ce33f5a-ba7b-11e7-9abd-52540065bddc
https://testing.hpdd.intel.com/test_sets/336e7196-12b9-11e8-a10a-52540065bddc
https://testing.hpdd.intel.com/test_sets/d61ba008-167e-11e8-bd00-52540065bddc

We see the other sanity 300* tests fail in similar ways and can open separate tickets if we think the following have distinct root cause for the failures.

sanity test 300d fails with

== sanity test 300d: check default stripe under striped directory ==================================== 06:31:06 (1516775466)
CMD: trevis-13vm4 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
open(/mnt/lustre/d300d.sanity/striped_dir/f1) error: Permission denied
total: 1 open/close in 0.01 seconds: 189.21 ops/second
 sanity test_300d: @@@@@@ FAIL: create 10 files failed 

with logs at
https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc

sanity test 300e fails with

== sanity test 300e: check rename under striped directory ============================================ 01:22:13 (1519521733)
CMD: trevis-66vm8 /usr/sbin/lctl get_param -n version 2>/dev/null ||
				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/a': Permission denied
touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/c': Permission denied
mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_a': Permission denied
mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_c': Permission denied
rename returned -1: No such file or directory
 sanity test_300e: @@@@@@ FAIL: rename dir under striped dir fails 

with logs at:
https://testing.hpdd.intel.com/test_sets/cceb366a-19df-11e8-a6ad-52540065bddc
https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc
https://testing.hpdd.intel.com/test_sets/4b90ec14-ffb7-11e7-bd00-52540065bddc

Issue Links

• is duplicated by: LU-11487 sanity: 300b failed with 'touch error 0'
• is related to: LU-10592 sanity test_300h: create files failed
• is related to: LU-13400 sanity test_300d: createmany 10 under striped dir fails with Permission denied

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40585/
Subject: LU-10753 osd-zfs: initialize obj attr correctly
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 201efe7eea3f7d6fb6410db8b78d0c9187c274ce

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40585
Subject: LU-10753 osd-zfs: initialize obj attr correctly
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: aa62d909598d89d13bf564680ca2258daa1c0edf

[Olaf Faaland]

Lai,

Should this be backported to b2_12?  It also uses la_flags without first checking la_valid.

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40062/
Subject: LU-10753 osd-zfs: initialize obj attr correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cf395c2507e80717e7468456e9959d432b6accc8

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40062
Subject: LU-10753 osd-zfs: initialize obj attr correctly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2a89e9090b57a333e9bff89ce34f315f99ff49f1

[Chris Horn]

+1 on b2_12: https://testing.whamcloud.com/test_sessions/48af3de1-6698-480f-bf8b-9c03a0777418

[Emoly Liu]

more on master:
https://testing.whamcloud.com/test_sets/53081ec8-1310-4cc9-9215-e17a0ce8a432
https://testing.whamcloud.com/test_sets/887d2132-33fe-4a5e-b850-5c5ae64107de

[Chris Horn]

+1 on master https://testing.whamcloud.com/test_sessions/4496d6f6-4562-485b-a62f-51bf7bd2b14a

[Bruno Faccini (Inactive)]

+ 1 with recent master at https://testing.whamcloud.com/test_sets/21a96f11-1563-4baa-9f9e-9b3dbb59f19e