[LU-8118] very slow metadata performance with shared striped directory

Tue, 7 Jun 2016 15:38:11 +0000

We are experiencing a severe metadata performance issue intermittently. So far it has occurred with a striped directory being alter
ed by many threads on multiple nodes.

Setup:
Multiple MDTs, one MDT per MDS
A directory striped across those MDTs with lfs setdirstripe -D set
several processes on each of several nodes making metadata changes in that striped dir

Symptoms:
Create rates in 10s of creates/second total across all the MDTs hosting the shards
getdents() times >1000 seconds per getdents() call

In one case:
The directory was striped across 10 MDTs and 16 OSTs
Mdtest had been run as follows:

srun -N 10 -n 80 mdtest -d /p/lustre/faaland1/mdtest -n 128000 -F

The create rate started out about 30,000 creates/second total across all 10 MDTs. After some time it dropped to 10-20 creates/secon
d. On a separate node, which mounted the same filesystem but was not running any of the mdtest processes and was entirely idle, I r
an ls and observed the very slow getdents() calls.
On yet another idle node, I created another directory, striped across the same MDTs, and created 10000 files within it. Create rate
was good. Listing that directory produced getdents() times of about 0.003 seconds.
There were no indications of network problems within the nodes at the time, nor before or after our test (this is on catalyst and th
e nodes are normally used for compute jobs and monitored 24x7).

In the other case:
This filesystem has 4 MDTs, each on its own MDS, and 2 OSTs.
The directory is striped across all 4 MDTs and has -D set.

The workload involved 10 clients, each running 96 threads. Shell scripts were randomly invoking mkdir, touch, rmdir, or rm (the lat
ter two having chosen a file or directory to remove).

Create rate started at about 10,000/sec concurrent with thousands of unlinks, mkdirs, rmdirs, and stats per sec. All those operatio
ns slowed to single digit per-second rates. An ls in that common directory, on a node not running the job, also produced >1000 seco
nd getdents() calls.

strace -T output:

getdents(3, /* 1061 entries */, 32768)  = 32760 <1887.749809>
getdents(3, /* 1102 entries */, 32768)  = 32760 <1990.174707>
getdents(3, /* 1087 entries */, 32768)  = 32752 <1994.781547>
getdents(3, /* 1056 entries */, 32768)  = 32768 <1907.404333>
brk(0xcf0000)                           = 0xcf0000 <0.000030>
getdents(3, /* 1091 entries */, 32768)  = 32752 <1860.720958>

Whamcloud Community JIRA

[LU-8118] very slow metadata performance with shared striped directory