[LU-9666] sanity test_51f: create+open 100000 files in /mnt/lustre/d51f.sanity failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Jian Yu
• Resolution: Won't Fix

Environment

Servers/CLient : master, 2.9.58, 3591

• Severity: 3

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/4997e0d2-4af1-11e7-91f4-5254006e85c2.

The sub-test test_51f failed with the following error:

create+open 100000 files in /mnt/lustre/d51f.sanity failed

test logs:

== sanity test 51f: check many open files limit ====================================================== 20:08:40 (1496261320)
MDT0 numfree=522481, max=100000
changed ulimit from 1024 to 100020
 - open/keep 10000 (time 1496261329.90 total 9.41 last 1062.51)
 - open/keep 20000 (time 1496261339.42 total 18.93 last 1050.59)
 - open/keep 30000 (time 1496261348.90 total 28.41 last 1054.53)
 - open/keep 40000 (time 1496261358.71 total 38.22 last 1019.66)
 - open/keep 49787 (time 1496261368.71 total 48.22 last 978.68)
 - open/keep 50000 (time 1496261368.93 total 48.44 last 979.94)
 - open/keep 58551 (time 1496261378.93 total 58.44 last 855.09)
 - open/keep 60000 (time 1496261380.60 total 60.11 last 866.23)
open(/mnt/lustre/d51f.sanity/f65159) error: No space left on device
total: 65159 open/keep in 65.33 seconds: 997.38 ops/second
 - closed 10000 (time 1496261389.94 total 4.12 last -12137.58)
 - closed 20000 (time 1496261393.91 total 8.09 last 2518.69)
 - closed 30000 (time 1496261398.06 total 12.24 last 2411.40)
 - closed 40000 (time 1496261402.20 total 16.38 last 2414.27)
 - closed 50000 (time 1496261406.33 total 20.51 last 2419.03)
 - closed 60000 (time 1496261410.49 total 24.67 last 2403.74)
total: 65159 close in 26.72 seconds: 2438.76 close/second
 sanity test_51f: @@@@@@ FAIL: create+open 100000 files in /mnt/lustre/d51f.sanity failed 

Seen only for upgrade/downgrade testing.

Comments

[Peter Jones]

Jian

Can you please investigate?

Thanks

Peter

[Jian Yu]

Hi Saurabh,
The real failure is:

open(/mnt/lustre/d51f.sanity/f65159) error: No space left on device

Dmesg on OST:

LustreError: 8749:0:(ofd_dev.c:1775:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28

OST was out of space. Many tests in the test session failed due to this.