|
Stephane,
I won't be able to examine them immediately, but dmesg and the output of lctl dk [you don't have extra debug on, but it still includes additional info vs dmesg] from the MDS(es) would be useful.
Just to be clear:
Do these timed out messages specifically correspond to clients you're rebooting? ie the NID is that of a client that was being rebooted at that time?
|
|
Thanks Patrick, I'll try to gather more logs a bit later. But to answer your question, yes this specific node was redeployed around 13:26:43 and the MDT logs with timestamps are:
Not sure at all this is directly related to our hang issues but that's the only messages I could see at that moment.
|
|
Are you unmounting the fs gracefully on the nodes you're rebooting?
If they're doing certain types of activity and go away suddenly, the MDS can end up needing to call back locks from them, which means it has to time out requests and evict them, and until that happens, the system will be hung. This sounds rather like that.
|
|
Patrick, we use xCAT and rinstall should unmount the fs gracefully for normal redeployment, but we also have nodes that end up in a bad state sometimes that we have to reset, this cannot be avoided. At this time, it seems that the frequency of the hangs have diminished during the last 24h. Perhaps a good effect of the progressive deployment of the patch for LU-11964, as the lru_size=clear workaround was perhaps not 100% proof-safe, I'm wondering. I'm not willing to touch anything at this time during a weekend but will let you know if I have more information. Thanks!
|
|
Hi Patrick,
This morning at 8:09 AM:
At that the exact same time 8:09, we have this on one MDS (fir-md1-s1):
The corresponding node is sh-20-15 (10.8.20.15@o2ib6), and we confirmed that this node crashed probably due to some hardware issue. We don't see this issue with 2.10 or 2.8 filesystems.
I was able to gather some Lustre logs from the MDS, not sure they will reveal something though, but it's worth a shot. I'm attaching the logs as fir-md1-s1_dk20190225.log.gz
Thanks!
Stephane
|
|
Can you do fid2path on [0x200000400:0x5:0x0]? It may not work.
After fid2path, please also do lfs getstripe on that file, if you can find the file.
|
|
It's one of our base directory:
As a FYI, it uses DNEv2 stripping to autoselect one of the MDT when a new homedir is created:
|
|
That way, we get scratch home dirs balanced on our 4 MDTs:
Just as a FYI, don't know if that is related. 
|
|
Ah, OK. Interesting. OK, here's the general outline of the problem.
The client is holding a pretty standard lock on that directory, the kind that suggests it did a lookup under it. (May be other operations as well, but lookup at least)
Some change in 2.12 - possibly DNE2 related - means that some other operation needs to cancel that lock so it can proceed. So when a client goes away ungracefully, these other operations have to wait for this eviction. (BTW, this strongly suggests your intentional reboots to install the patch were indeed not unmounting the fs.)
Hmm. I can probably reproduce this with a little effort.
Did you use DNE2 in your pre-2.12 install? (And what version of Lustre was that install?)
|
|
Patrick, OK! Just before putting /fir into production, we formatted it with 2.12 GA, so it's a new 2.12 filesystem. We only use DNEv2 in that specific case for /fir/users/ (all users) and /fir/groups/ (all labs/groups).
|
|
So you have not used DNE2 in a similar way on a 2.10 system? (Trying to bound when the issue might've started )
|
|
That's correct, we started to use DNE2 with /fir in 2.12 only.
|
|
If this happens again, can you dump the lock table on the MDS during the hang?
|
|
Sure, will do!
Thanks!
|
|
Stephane,
After some careful examination, I'm not sure this is a new problem. If you've got a client that has done a lookup on the and it is lost, then creating a new file in another directory can hang. I can reproduce this on 2.10 without any DNE being involved.
I don't have a 2.7 system handy right now...
I can reproduce this with two clients and no DNE.
One client does ls -l in a directory, and then panic that client. The other client tries to create a file in a subdirectory of that first directory (remember that the first client did ls in the parent). This hangs until the eviction is complete. (It also works if the ls and the create are in the same directory.)
I believe I've seen this behavior in older systems as well, so I don't think it's new.
laisiyao, can you take a look at this and confirm what's supposed to happen here?
|
|
More issues tonight that started with directories not accessible from a specific client (10.9.0.1@o2ib4) and ended up with a full filesystem deadlock. Restarting all MDTs resolved (temporarily) the issue...
This is the console of fir-md1-s1 (serves MDT0000 and MDT0002), when the directories started to become unaccessible:
MDTs restarted occurred at ~15:47
I was able to gather some logs:
- a tar of the lustre-log files above (automatically generated) is available in the FTP (fir-md1-s1-lustre-log-20190228.tar.gz)
- I used the commands that Patrick provided above, I did it two times, the results is available in fir-md1-s1-20190228-1.log.gz and fir-md1-s1-20190228-2.log.gz attached to this ticket
- fir-md1-s1 full kernel syslog dump in fir-md1-s1-kern-syslog-20190228.log
Thanks!
|
|
Huh! Stephane, this is a different issue. Can you open a ticket for this with these logs? Looks like a DNE bug of some kind - the MDTs are having connection problems between one another.
|
|
Doh. Sure, done: LU-12037
|
|
To come back at our original issue, it happened again today, a node crashed (or rebooted, I'm not sure), anyway, the whole filesystem was blocked for a ~2 minutes, including the Robinhood server (scan completely stopped).
This happened today between Mar 04 15:58 and Mar 04 16:00
MDS fir-md1-s2 MDT0001 / MDT0003 logs:
I'm not sure why the other MDT, fir-md1-s1 (MDT0000/MDT0002) didn't complain at all.
I understand that locking could do such thing, but we've never seen that with previous version of Lustre, and this stops ALL metadata operation on the filesystem, globally.
We monitor md ops on the MDTs, and I'm attaching a screenshot of Grafana showing the global drop. Not good... Thanks!
|
|
Again, filesystem blocked while a node crashed
During that time, we couldn't ls anywhere on /fir (for about 2 minutes, tested from several nodes and subdirectories). Is there a timeout we could lower to mitigate this?
|
|
Stephane,
Are you able to dump the lock tables? Remember that when you did that before, it was actually that DOM/rename related hang (LU-12037), not this issue. Having the full table would be very helpful.
I will take a look at timeouts, this should be adjustable.
|
|
Hi Patrick!
We have Lai's patch for LU-12037 applied on both MDS now, so...
I'll try to dump the lock tables next time this happens. Thanks!
|
|
Wasn't very long... the client was 10.8.20.15@o2ib6 in that case:
fir-md1-s1 (MDT0000, MDT0002):
fir-md1-s2 (MDT0001, MDT0003):
I think I got the MDT lock table dumps during that time! Filesystem just unblocked while I was finializing dumping it on fir-md1-s2. Attaching fir-md1-s1_dlmtrace_20190312.log.gz and fir-md1-s2_dlmtrace_20190312.log.gz. Thx.
|
|
Stephane,
Have you changed any settings related to adaptive timeouts (AT_MIN, AT_MAX) or obd_timeout? (Not suggesting you change these yet - Just asking.)
|
|
And thank you for the logs, will investigate.
|
|
Would you be able to tar up and attach dmesg from all your MDSes?
|
|
Stephane,
Looking in to the timeouts, you can tweak them, but it's a pretty big hammer. We don't want to set these too low or it may cause trouble.
Let's aim for 30 seconds.
On only the MDSes.
You need to set ldlm_enqueue_min to 30, then at_max to 20. (1.5*at_max is the MDS bl_ast timeout. Both are ptlrpc module parameters).
This should reduce your hang time from 150 seconds to 30, and 30 should still be plenty of time.
|
|
Thanks!
Added that to our /etc/modprobe.d/lustre.conf:
And restarted MGS +MDTs.
|
|
LU-12064 has been opened to look at possible adjustments to the adaptive timeout code. Note this is not a fix for this issue, just something to hopefully reduce delays.
|
|
Just wanted to follow-up on this one, as today we noticed the same issue against regal, our 2.8-based old scratch filesystem, mounted under /regal, after rebooting a few compute nodes. We have never seen this behavior before upgrading our clients to 2.12, so we strongly suspect some regression there. It would be weird that we missed that kind of behavior for several years (and it's not like we're not rebooting nodes ). And there is no DNE, DoM, PFL involved there.
Server (2.8) logs:
My colleague had that following command blocked during that time:
[root@sh-hn01 regal]# mkdir /regal/.deleted/hasantos/
But no logs on the client (2.12).
Access to /regal was restored after a few minutes.
|
|
We still had the issue this morning, this was when Kilian was creating a bunch of new accounts (mkdir) along with rebooting nodes. mkdir was done in /fir/users and /fir/groups which are DNE2 enabled.
Not everything was blocked, but find in /fir/users and /fir/groups was hanging on all compute nodes. However, I was able to browse /fir/users/sthiell/ without issue (DNE1 only).
Attaching full kernel logs from both MDS as fir-md1-s1-20190508.log and fir-md1-s2-20190508.log. The issue happened around May 08 11:23 in the logs which can particularly be seen in fir-md1-s2-20190508.log
After a few minutes, things went back to normal.
|
|
This issue is still ongoing, and still seems to happen when we use mkdir with DNEv2 (each time we hit this issue there is a mkdir using DNEv2 running on sh-hn01 (10.9.0.1@o2ib4)...
The full filesystem was blocked again today for more than 10 minutes while a mkdir was running there. There are some backtraces like that when things started to recover:
I'm attaching MDS logs `fir-md1-s1` as fir-md1-s1_20190715.log (server where is hosted MDT0000, MDT0002 and the MGT).
WARNING: this is the full kernel logs since last MDS restart, June 18, 2019!!! Not too bad for 2.12!! Our event started on July 15, 2019 at about 15:20 in the log. First user ticket arrived at 15:24.
The mkdir that was blocked during this event was on the following directory, automatically stripped on MDT0, which eventually succeeded:
As a reminder:
The good thing is that the MDS recovered by itself but it took some time.... at least 10 minutes.
Thanks,
Stephane
|
|
I've been observing something similar when we have clients that mistakenly associated with the MDS using an NID behind a NAT. In this example, nodes came up with lnet binding to a private cluster interface instead of an infiniband interface.
With a lustre 2.10 client and 2.12 server, we can reproduce this behavior easily. The file system hangs for all clients. A relevant snippet from our logs is included - let me know if I can provide any additional information.
|
|
Stephane, are you still seeing these long hangs? Asking because we aren't yet using DNE2 but would like to (originally intend to with Lustre 2.8 but decided against it, and now revisiting the idea).
|
|
Hi Olaf – No, we haven't seen such long hangs for quite a while now, which is a good thing. They were very frequent when using mkdir with DNEv2, but the issue seems to have been resolved. We're still using 2.12.3 + a few patches (server-side) but they that have all landed in 2.12.4.
|
Generated at Sat Feb 10 02:48:42 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
fir-md1-s1-20190228-1.log.gz
fir-md1-s1-20190508.log
fir-mdt-grafana-fs-hang_mdt1+3_20190304.png