[#LU-2932] slow metadata performance at NOAA again

[LU-2932] slow metadata performance at NOAA again Created: 08/Mar/13 Updated: 09/May/14 Resolved: 09/May/14
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.7
Fix Version/s:	None

Type:

Bug

Priority:

Major

Reporter:

Oz Rentas

Assignee:

Cliff White (Inactive)

Resolution:

Won't Fix

Votes:

Labels:

None

Attachments:

scr1mdtest.xlsx

scr2mdtest.xlsx

version_patches_b1_8_patches_increase-ldlm-hashbuckets.patch

Severity:

Rank (Obsolete):

7049

Description

During the last downtime at NOAA, we discovered that the slow metadata performance reported in ~~LU-2649~~ had returned. The strange thing is that when the clients were shutdown, the performance returned to normal. The current theory is that the jobs that run on one filesystem are taking more/different locks than the jobs on the other. We are taking a downtime next Thursday to investigate this more, but would like some pointers as to what we can look for. Does this theory make sense?

Comments

Comment by Cliff White (Inactive) [ 08/Mar/13 ]

If you are taking a downtime, it would be good to try and isolate the problem as much as possible, eliminate any possible background load. If you can dial down to the smallest reproducer it will help. You might run some baseline tests, such as lnet_selftest to verify that other factors such as network are still equal between the systems.

From the last bug, i don't think you tried increasing mdt threads, that may be something to look at.
If the jobs are accessing data of the same size, in the same manner, and inside the same sort of file/directory structure, the locking shouldn't be much different. You could trace ldlm traffic, but the hit from enabling debug would likely obscure the result.

It might be good to strace some jobs, and see if you can find where the the time is going (-T, or with some version -ttt) or if there are other differences between the jobs. Can you run the exact same task on both filesystems?

Comment by Cliff White (Inactive) [ 08/Mar/13 ]

Again, from the previous bug, looking at disk hardware performance on the mdt would be good. sestets, iostat, or even sar would perhaps show something.

Comment by Kit Westneat (Inactive) [ 08/Mar/13 ]

Hi Cliff,

We've pretty much exhausted the hardware tests. It's really bizarre that simply shutting down the clients causes performance to go up significantly. I should mention that before we shut them down, we have looked at the export stats to verify that there is absolutely no load on the system. Even with no load at all (all the stats file remaining constant), the performance is degraded until the clients are shutdown, and therefore evicted from the system. When the clients are put back into production, performance starts slowly going down. After a day or two, it's back to the "bad" performance numbers. This, coupled with the dlmtraces from the last bug, makes me think it's some kind of issue in the lock manager that gets cleared when the clients and their locks are evicted.

As mentioned in the previous bug, we have been testing exclusively with mdtest. We hope to use this downtime to discover what applications or operations cause the filesystem to go into degraded performance mode.

Thanks,
Kit

Comment by Nate Pearlstein (Inactive) [ 08/Mar/13 ]

I think running perf with the same tests to show where the kernel is
spending its time would be helpful as well. It should be less expensive
than dlmtraces and well give a more global picture of what is going on.