|
The mdtest results would point toward MDS/network. I believe req_waittime for llstat oat is the time for the IO in the disk queue. I will ask our networking people for further advice.
|
|
They have suggested using LNET self-test to determine if the routing differences are significant. A test such as:
"lst add_test --batch bulk_rw --concurrency 16 --from client --to server brw write size=1M"
would be useful, where 'client' and 'server' are a client and the MDS node.
See the Lustre Manual for lnet self-test example scripts if you haven't used if before.
|
|
LNET selftest actually performed better on the scratch2 (slow) MDT than on the scratch1 MDT to the same client. scratch2 was 2.9-3GB/s and scratch1 was 2.8ish. I tried doing a single rpc in flight test too, in order to see if latency was an issue, but they both ran at the same speed, 1.5-1.6GB/s. I tried doing a 4k test and RPC rates were lower on scratch2 by about 10% with concurrency 8. When I lowered the concurrency to 1, I got 30k rpcs on scratch2 vs. 40k rpcs on scratch1. That is somewhat significant, but it still not a large enough difference to explain the giant dirstat differences.
|
|
Any more ideas for things to check? The metadata performance delta between the filesystems is still fairly significant and we still haven't narrowed it down to any component.
|
|
Just to throw another ? into the equation. I looked at import on scratch1 and scratch2 right after running mdtests on both, and they both say 31s as their service estimates.. Though scratch1 has a waittime of 182ms vs a waittime of 262ms for scratch2.
|
|
Looks like it's a network problem between MDT to OSTs (and looks the connection between client and OSTs aren't good as well), I think we'd run LNET selftest between MDTs and OSTs but not the client to MDTs.
|
|
I ran MDS <-> OSS testing with lnet_selftest. There was a lot of variation in the numbers due to the active workloads on them (the system is in production), but I couldn't see a significant difference between the scratch1 filesystem and scratch2. I will try to get better data, but at the moment the LST data is inconclusive.
I was looking at more routes and it looks like some of the scratch1 routes are asymmetric as well, so I don't think that can be pointed at as the root cause. I am still seeing a 2x avg req_waittime between the MDT and OSTs on scratch2 vs scratch1 though. I'll try to look at the individual OSTs to see if I can see a pattern.
|
|
Hi Niu,
I was wondering if you could expand on what makes it seem like a networking issue as opposed to anything else. The customer is interested. Also, is there anything else we could check to definitively rule out any other issues?
Thanks,
Kit
|
|
Question: have the tunable LND parameters (credits, peer_credits, etc) been modified on any of these systems? To help see if credits are playing a role in throttling at the LNet layer, can you post the results of these two "cat" commands on the systems with potentially slow networking:
cat /proc/sys/lnet/nis
cat /proc/sys/lnet/peers
cat /proc/sys/lnet/stats
These three proc files will show information about credits as well as potential networking errors.
Are there LNet routers involved? If so, can you post the results of this "cat" command on those routers:
cat /proc/sys/lnet/buffers
That one will show if we are running out of routing buffers thereby creating a bottleneck.
|
|
To some extent, we are going by you initial problem statement, you said you checked the storage, and it appeared to be performing properly. In the case of the mdtest results you reported, those depend quite a bit on network/MDS storage performance. Anything further you can check related to storage performance would help eliminate that as an issue.
|
|
Ok, I'll attach that output from OSSes and MDSes on scratch1 and scratch2.
In getting this info, I realized that I had been testing the wrong fabric before for the OSS <-> MDS tests. The LST tests again looked pretty decent except for when I did the MDS as the server, the OSS as the client and did a write-only test. Then it bounces between 1.2-2.2 GB/s (most of the time around 2GB/s though). If I did the MDS as the client, and/or did a read-only test, the results were around 3GB/s. This behavior is seen on both scratch1 and scratch2 filesystems.
The IO to disk looks decent, but I was wondering if there was some way to rule out the layers above the disk. Is there a way to see how long IOs are waiting in Lustre's queues before going to the disk? Is it possible for ldiskfs to add significant time? Is there anything else we can check?
|
|
lnet data
|
|
In those collected LNet stats are a lot of negative "min credits". That represents the low water mark for credits for either a network interface or peer (depends on the proc file being viewed). A negative number indicates that queuing has been going on and shows how deep the queue became.
This indicates to me that some level of throttling is going on. I cannot say for sure that this is enough to create bandwidth reductions you are seeing. One way to find out is to bump up the credits and peer_credits on critical nodes like the MDS and OSS's. See the Lustre manual on the module parameters used to modify the credits and peer_credits.
|
|
One thing I noticed is that the OSS has a smaller min for the MDS peer line for scratch2 (-14) than for the scratch1 counterparts (-421). What does this mean? Does this mean that there is less queuing on the scratch2 network than scratch1?
|
|
That is correct. There seems to be less queuing on the scratch2 network.
|
|
Hmm, would that imply that the network is most likely not a factor in scratch2 being slower? If there are not many requests waiting in queue, it seems like the problem is in processing the requests and getting them into the queue.
We have made some progress.. Cpu speed regulating was enabled on the OSSes and MDSes, turning that off has improved performance across the board. There is still a large performance difference between scratch1 and scratch2, of the same magnitude as before.
Are there any other stats we should check?
|
|
I did a trace and rpctrace debug dump. It looks like the time delta is almost entirely in the ldlm code. Do you have any thoughts as to what might cause that?
I am going to try to enable ldlm debugging and see if I can get any more information.
|
|
Can you attach/upload the trace data?
|
|
Also, have you tried adjusting the peer credits, as Doug mentioned above?
|
|
rpctrace and trace of a single request with timestamp deltas added
|
|
The customer won't take a downtime to change the credits unless I can say that I think there is a high probability of solving the issue, and I don't have that level of confidence right now.
|
|
dlm traces, I'm not sure if these actually add anything
|
|
Hello, any updates on this issue? I have been having regular status calls about it, so any ideas would be very helpful in keeping the customer content.
|
|
I am talking to our engineers, hoping to have some ideas for you soon.
|
|
It seems pretty clear that entire MDS operations on one filesystem are slower than on hte other and it's not affected by OST performance (see how dir operations are also degraded and those don't really touch OSTs).
I wonder if the load on the more slow MDS is just different and as such leads to slower performance for other clients? (I assume there's a constant load in background in there?)
Another interestign thing would be to compare rpcstats and sdiostats from the two MDSes to see how those differ, I think
|
|
There is a variable background load, to be sure. scratch2 is often more loaded, but I've tried to run the tests when there isn't very much activity on the MDSes (as seen by the export stats). I'll try to get a more precise measurement of relative activity via rpcstats. When you say sdiostats, which stats in particular are you thinking of? I've used iostat to watch the disk activity and I don't see a lot of differences between the two.
One thing that occurred to me is that scratch2 has 20 OSSes, while scratch1 only has 16 (and scratch2 has 25% more OSTs as well). Could the time spent in the LDLM code be proportional to the number of OSSes/OSTs? It would explain the 25% reduction in performance, to be sure.
|
|
LDLM on MDS does not deal with OSTs at all. The only place on MDT that does is in LOV (and OSCs) to do object pre-allocations (also setattrs and such).
Also when doing directory creates/removals, OSTs are not contacted at all, so the only impact they could make is if there's a lot of say creations going in background that wait for ost precreations and block other threads from executing, but it's unlikely to be happening all the time, other mds activities could have similar impact.
What's the typical mds wqueue waittimes before requests are serviced?
I am interested in iostata data about typical io completion times and io sizes.
|
|
I looked at the /proc/fs/lustre/mds/*/stats and the /proc/fs/lustre/mdt/MDS/mds/stats files to see how loaded the system was. scratch2 does seem more loaded, but it might have just been when I was looking. I tried to find a moment when scratch2 was less loaded in order to run the benchmark. The mdtest showed some improvement but the dirstats were still significantly lower than on scratch1. Clearly the load on the system has some effect, and there is a downtime next week we'll use to test.
What sorts of tests would be useful on a quiet system? I was going to enable various debug flags in order to be able to hopefully trace out what is going on. Is there anything else you would like to look at?
What could cause dirstat performance to be so much worse than the other metadata operations?
Here are some example req_* llstat mds output:
scratch1:
/proc/fs/lustre/mdt/MDS/mds/stats @ 1359617590.869072
Name Cur.Count Cur.Rate #Events Unit last min avg max stddev
req_waittime 14886 7443 4161976 [usec] 300812 3 18.38 3017 14.82
req_qdepth 14886 7443 4161976 [reqs] 1870 0 0.18 14 0.44
req_active 14886 7443 4161976 [reqs] 22011 1 1.89 55 1.59
req_timeout 14886 7443 4161976 [sec] 44658 3 26.59 31 10.20
reqbuf_avail 31265 15632 9080719 [bufs] 96035468 3044 3071.47 3072 1.07
ldlm_flock_enqueue 5 2 3190 [reqs] 5 1 1.00 1 0.00
ldlm_ibits_enqueue 14199 7099 3808598 [reqs] 14199 1 1.00 1 0.00
scratch2:
/proc/fs/lustre/mdt/MDS/mds/stats @ 1359617591.258278
Name Cur.Count Cur.Rate #Events Unit last min avg max stddev
req_waittime 18790 9395 22080943 [usec] 389637 3 21.72 103629 171.90
req_qdepth 18790 9395 22080943 [reqs] 5206 0 0.33 147 0.92
req_active 18790 9395 22080943 [reqs] 64994 1 5.17 127 12.89
req_timeout 18790 9395 22080943 [sec] 93950 2 15.56 33 14.32
reqbuf_avail 44790 22395 53001699 [bufs] 894286323 19850 19965.41 19968 8.94
ldlm_flock_enqueue 38 19 19939 [reqs] 38 1 1.00 1 0.00
ldlm_ibits_enqueue 13264 6632 15674307 [reqs] 13264 1 1.00 1 0.00
I was wondering why the reqbuf_avail on scratch2 was so much higher?
|
|
IO statistics for the sd devices, they look very similar between the FSes
|
|
differences in /proc/fs/lustre/mds/scratch2-MDT0000/stats over the run of an mdtest. You can see scratch2 was much more loaded during this time. The opens, closes, and unlinks stick out as major differences too.
|
|
I've attached the test plan for the downtime next week, can you review it and see if there is anything else we should be looking at or trying?
|
|
What about the peer credits tuning mentioned earlier?
|
|
What are the risks in making that change and how likely do you think it is that those changes will affect performance?
|
|
I think the risks are minimal, shouldn't hurt anything. I am not certain how likely it is that it will help this issue.
|
|
Ok, the reason I ask is that any changes have to be approved through a change review meeting, and I'm sure that they will ask me those questions.
Actually another question I have is: can those be set only on the servers, or do they need to be set on the clients too? I remember the last time I played around with ko2iblnd settings, there were some that had to be set on both the client and the servers, or they would stop communicating.
|
|
Credit changes affect the "throttling" on the system they are set on. I suspect the important systems to change here are just the servers. The clients (assuming there are many of them) should be fine as is. The changes don't need to be the same on both sides of a connection.
|
|
Note if you enable any extra debug flags, you'll likely have (significant, depending on flags) slower metadata performance for certain debug flags like +inode or +dlmtrace.
It looks like scratch1 is almost 2x less loaded than scratch2 at times, and I suspect that on the fully quiet system you'll have same level of performance.
dirstat is only affected by network latency and how busy MDS/disk is (even then I suspect for mdstat run created directories are fully in cache and disk does not play any role other than when it slows down other threads).
I suspect that if you forcefully increase number of mdt threads (module parameter for mds module) then dirstat should go up if you are running with a small number by default.
|
|
It seems like credits changes need to be made on all systems (clients and servers).
I got these errors trying to ping from a node with credit changes to one without:
LustreError: 9688:0:(o2iblnd.c:806:kiblnd_create_conn()) Can't create QP: -22, send_wr: 16448, recv_wr: 256
LustreError: 7641:0:(o2iblnd_cb.c:2190:kiblnd_passive_connect()) Can't accept 10.175.31.242@o2ib: incompatible queue depth 8 (128 wanted)
|
|
Looks like I was wrong about the "credits" tuneable. For IB, it controls the QP depth which should be symmetric (and we ensure it is symmetric as seen in the error messages). All the systems on the same IB network (could be logical networks) need to have the same credits setting.
|
|
we can close this. We found a potential microcode issue with the Intel chipset and upgraded that. We also split the two filesystems over different fabrics. Metadata improved on both filesystems.
|
|
ok thanks Kit!
|
Generated at Sat Feb 10 01:27:01 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
dlm_trace.scratch1
test_plan.txt