Configuations
==========
|
|
Just to add to this - although the filesystem isn't completely 'down' it is interrupting filesystem access to a large portion of our clients so I marked this as priority '2' instead of '1'. However it is now a serious problem for us as the issue keeps reoccurring, so if it's possible at all to get time with anyone to look at this together, it would be greatly appreciated? I can offer a shared remote session via SSH if that's useful at all, or available for zoom/teams call too.
|
|
Attached are kernel logs from one of the MDS servers and one of the LNET routers from today. Both have shown multiple signs of the symptoms mentioned earlier today. One particular timeperiod of such problems were between 10:00 and 12:00.
mds.log router.log
Not sure if related, but I also noticed the following crash on one the same MDS I've taken the logs above from, multiple times a day at the moment:
Am I right in relating this to changelog records filling up/overflowing on that MDT? I will try clearing them and see if the error goes away.
The lustre log dump from this event is here:
lustre-log.1630947202.26758.gz
|
|
Serguei
Can you please advise?
Thanks
Peter
|
|
Hi,
I can't comment on the MDS issue with logging, maybe Peter can get someone to look at it.
The combination of
retry_count: 2
transaction_timeout: 50
may result in transaction timeout being too short: 50/(2+1) per attempt. This can cause the nodes to give up on the transaction and retry too early, thus generating extra traffic in an already busy system.
I'd recommend increasing transaction_timeout to 150 on all nodes. You can change it without rebooting the nodes via lnetctl, but to make the setting persistent, it would have to be added via the conf files.
In order to check the routing configuration, could you please provide the output of
from a typical client from each of the networks (opa, mlnx, tcp) and the server?
We can setup a zoom call today at any time that works for you, it will be useful for collecting debug logs and verifying configuration at this point.
Thanks,
Serguei.
|
|
Thanks Serguei - the retry_count issue came up in another ticket recently actually - I went into this issue with the servers, only, using transaction_timeout: 150 and retry_count: 3, but I reset those back to the defaults early on in a fruitless attempt to standardise settings everywhere to try to get a handle on things.
Is the transaction_timout specific only to LNET messages sent by each peer - so only changing one end of a transaction isn't helpful, as it will only impact messages sent by one end of the interaction?
I'll work to standardising with the larger transaction timeout everywhere as recommended.
Regarding the zoom call - yes this would be very helpful, I can do anytime between 20:00-00:00 BST if you have time then?
I will gather the routing configuration shortly and post in a separate comment.
Kind regards,
Matt
|
|
Matt,
Yes, transaction_timeout setting needs to be the same across nodes in the system.
We can have the call at 20:00 BST (I believe it is noon PST?) Please post the meeting link here or send it to ssmirnov@whamcloud.com
Thanks,
Serguei.
|
|
Thanks - I've just sent an email to you with the invitation link in it.
|
|
Routing information:
route:
- net: tcp2
gateway: 10.44.240.167@o2ib2
hop: 1
priority: 0
state: down
- net: tcp2
gateway: 10.44.240.166@o2ib2
hop: 1
priority: 0
state: up
- net: tcp2
gateway: 10.44.240.161@o2ib2
hop: 1
priority: 0
state: down
- net: tcp2
gateway: 10.44.240.164@o2ib2
hop: 1
priority: 0
state: up
- net: tcp2
gateway: 10.44.240.165@o2ib2
hop: 1
priority: 0
state: down
- net: tcp2
gateway: 10.44.240.163@o2ib2
hop: 1
priority: 0
state: down
- net: tcp2
gateway: 10.44.240.162@o2ib2
hop: 1
priority: 0
state: up
- net: tcp2
gateway: 10.44.240.168@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.44.240.167@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib
gateway: 10.44.240.166@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.44.240.165@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib
gateway: 10.44.240.168@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.163@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.162@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.166@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.165@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.167@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.168@o2ib2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.161@o2ib2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.164@o2ib2
hop: 1
priority: 0
state: up
- net: tcp4
gateway: 10.44.240.162@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.166@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.167@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.163@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.164@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.165@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.161@o2ib2
hop: 1
priority: 0
state: down
- net: tcp4
gateway: 10.44.240.168@o2ib2
hop: 1
priority: 0
state: down
[root@cpu-p-350 ~]# lnetctl route show -v
route:
- net: o2ib
gateway: 10.44.240.165@o2ib2
hop: -1
priority: 0
state: down
- net: o2ib
gateway: 10.44.240.166@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib
gateway: 10.44.240.167@o2ib2
hop: -1
priority: 0
state: down
- net: o2ib
gateway: 10.44.240.168@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.168@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.165@o2ib2
hop: -1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.164@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.163@o2ib2
hop: -1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.162@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.167@o2ib2
hop: -1
priority: 0
state: down
- net: o2ib1
gateway: 10.44.240.166@o2ib2
hop: -1
priority: 0
state: up
- net: o2ib1
gateway: 10.44.240.161@o2ib2
hop: -1
priority: 0
state: down
[root@cpu-e-451 ~]# lnetctl route show -v
route:
- net: o2ib
gateway: 10.47.240.166@o2ib1
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.47.240.167@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib
gateway: 10.47.240.168@o2ib1
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.47.240.165@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.47.240.168@o2ib1
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.47.240.167@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.47.240.161@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.47.240.162@o2ib1
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.47.240.163@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.47.240.166@o2ib1
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.47.240.165@o2ib1
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.47.240.164@o2ib1
hop: 1
priority: 0
state: up
[root@cpu-f-1 ~]# lnetctl route show -v
route:
- net: o2ib1
gateway: 10.144.240.165@o2ib
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.144.240.167@o2ib
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.144.240.168@o2ib
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.144.240.166@o2ib
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.144.240.166@o2ib
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.144.240.168@o2ib
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.144.240.165@o2ib
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.144.240.167@o2ib
hop: 1
priority: 0
state: down
[root@rds-gate1 ~]# lnetctl route show -v 4
route:
- net: o2ib
gateway: 10.43.240.167@tcp2
hop: 1
priority: 0
state: down
- net: o2ib
gateway: 10.43.240.166@tcp2
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.43.240.168@tcp2
hop: 1
priority: 0
state: up
- net: o2ib
gateway: 10.43.240.165@tcp2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.43.240.163@tcp2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.43.240.166@tcp2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.43.240.167@tcp2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.43.240.162@tcp2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.43.240.168@tcp2
hop: 1
priority: 0
state: up
- net: o2ib1
gateway: 10.43.240.165@tcp2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.43.240.161@tcp2
hop: 1
priority: 0
state: down
- net: o2ib1
gateway: 10.43.240.164@tcp2
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.43.240.162@tcp2
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.43.240.165@tcp2
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.43.240.164@tcp2
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.43.240.168@tcp2
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.43.240.167@tcp2
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.43.240.163@tcp2
hop: 1
priority: 0
state: down
- net: o2ib2
gateway: 10.43.240.166@tcp2
hop: 1
priority: 0
state: up
- net: o2ib2
gateway: 10.43.240.161@tcp2
hop: 1
priority: 0
state: down
|
|
Hi Serguei, thanks again for your time today. The log files that we generated should be fetchable from our sftp gateway here using the key you sent me:
$ sftp whamcloud@vendor-gw01.hpc.cam.ac.uk
sftp> ls -1
cpu-f-1.debugging.20210907.log.gz
csd3-oss13.debugging.20210907.log.gz
csd3-oss14.debugging.20210907.log.gz
csd3-oss15.debugging.20210907.log.gz
csd3-oss16.debugging.20210907.log.gz
lnet1.debugging.20210907.log.gz
lnet2.debugging.20210907.log.gz
lnet3.debugging.20210907.log.gz
lnet4.debugging.20210907.log.gz
lnet5.debugging.20210907.log.gz
lnet6.debugging.20210907.log.gz
lnet7.debugging.20210907.log.gz
lnet8.debugging.20210907.log.gz
rds-oss31.debugging.20210907.log.gz
rds-oss32.debugging.20210907.log.gz
Let me know if you have any problems.
The client is 'cpu-f-1', all 8 routers are listed above, an interesting one to look at would be lnet6 perhaps, since that one is showing the odd message queuing behaviour at the moment. The 'fio' testing wrote to an OST on rds-oss31 above, so those three will hopefully contain a whole transaction to follow?
Kind regards,
Matt
|
|
Hi Serguei,
Unfortunately the situation for us has degraded fairly substantially over the last two days and we are now in a state where all clients are being affected at times due to LNET router-related instability.
During the last few days I was working to eliminate the remaining FDR based servers and clients from the network, so that we could reduce our routing configuration down to just: OPA (o2ib1) <–> EDR/HDR IB (o2ib2) and Ethernet (tcp2) <–> EDR/HDR IB (o2ib2).
This was completed on thursday with the last FDR servers moving to ethernet (tcp2), and we have just 5 remaining FDR (o2ib0) clients remaining, which we are working to also convert to ethernet in the upcoming days.
With this change, I took the opportunity on friday to change our LNET router configuration to the following setup:
So the thinking was to try to compartmentalize the configuration more, since previously we saw the message 'queuing' problems only on the routers dealing with FDR (o2ib0) in addition to the other networks, so I drew the hypothesis that this network was in some way responsible for the problem.
However we are now seeing the same problem on lnet[1-6] as of yesterday, attached here is a graph from yesterday evening showing the 'msgs_alloc' counter from 'lnetctl stats show' from these routers:
On the MDS servers at this time we see the following in the logs:
So this is starting shortly before the message queue spike on the LNET routers. We see this on all the MDS in the system and they all progressively get locked up, repeating this kind of stack trace repeatedly until they crash or are killed by pacemaker for being unresponsive.
Another similar looking stack trace from slightly later:
This is where we are at the moment. I've had to take all the routers offline for the moment to try and keep the MDS servers stable and allow our IB (o2ib2) clients at least access to the filesystems while I try to understand this latest development.
Have you had a chance to review the logs taken from our previous session at all? Do you have any time to have another call perhaps to look at this latest problem?
Thanks,
Matt
|
|
Hi Matt,
According to the router settings above you have discovery enabled. Have you tried to disable LNet discovery just on the routers?
I suspect LNet discovery to do some ping or message flooding. Our routers are not MR, only some compute nodes are. By disabling discovery on routers (discovery: 0 in lnet.conf), they are not getting completely overwhelmed and delaying traffic too much. This was a recommendation from Serguei for a problem I reported in LU-14652. We still often have queued messages though, but less.
|
|
Hi Matt,
I haven't finished going through the logs yet: I haven't seen any distinct signs of the problem in what I've seen there so far. We can schedule another call tomorrow. If the problem is more apparent now, at least we can collect more logs with the issue happening. I should be available from around 8:30 PST tomorrow (Monday). Please let me know here or via email which time works for you.
Thanks,
Serguei.
|
|
Hi Serguei,
This has become critical for our HPC service, over last week we had many disruption to production services, while this weekend we had to completely drain the jobs from the production clusters due to this issue. We had many customer complaining and need to find resolution quickly. As Matt said we have focused last week to make the configuration of LNET routing less complex by eliminating old unsupported hardware but so far things have not improved. Matt will arrange a call but I would appreciate if you bring appropriate engineering resource to help to resolve this very urgent issue for us.
Many thanks
Best regards,
Wojciech
|
|
Hi Serguei, thanks, best time for me is going to be from 11:30 PST onwards. I'll send you an email with a calendar invite.
@Stephane - Thanks, I will try this out. Do you disable discovery anywhere else, such as servers for example? Or even on clients too? Do you have to manually define peer information if you do this?
I've never really thought of our routers as MR, even though the peer information for a router marks it as that -
since each of these NIDs connects a different network, so they aren't redundant/dual connections in any way (what I usually think of as MR in lustre).
I have noticed some oddities in the peer definition for the routers across hosts, so I have been suspicious about this discovery mechanism - particularly with this new lockup on the MDS we are seeing inside 'lnet_discover_peer_locked'.
So I may try to statically define the router peer configuration next on the servers at least, and try with discovery = 0 on the routers and servers to begin with.
|
|
@Matt: I've only disabled Discovery on the routers. It is still enabled on all servers and clients. Nothing more, no need to manually define peer information either (I've been wondering about that but apparently, having it enabled on servers + clients seems to work for MR nodes – unless I missed something).
|
|
During the call, discovery was turned off on the routers. The following procedure was used:
1) Delete the routes from clients and servers. Delete router peers from clients and servers.
2) Bring down the routers, reconfigure (disable discovery)
3) Bring up the routers.
4) Add back routes on the servers
5) Add back routes on the clients.
One of the clients was rebooted, then was brought up and failed to mount. The debug trace from the client is attached. client.mount.debug.log
"lnetctl ping" to the nids used by the mount command was successful. lnet_selftest was also successful.
Clients that were not unmounted appear to be able to access the FS via the reconfigured routers.
Please provide the MGS client config log. I thought it was captured, but couldn't find it on the sftp server.
|
|
Amir pointed out that not only the client llog may contain bad NIDs, but server llogs as well, so they all need to be examined:
lctl --device MGS llog_print $fsname-client
lctl --device MGS llog_print $fsname-OSTxxxx
lctl --device MGS llog_print $fsnmae-MDTxxxx
If unreachable NIDs are listed, it is likely the cause of the "stuck discovery" trace that was observed, as Lustre will try to get the "primary NID" for these regardless of the discovery setting. If unreachable NIDs are listed in the client llog, it is likely the cause of the mount operation getting stuck.
If this is the case, my understanding is that the llogs will need to be regenerated: https://doc.lustre.org/lustre_manual.xhtml#lustremaint.regenerateConfigLogs
Thanks,
Serguei.
|
|
Hi Serguei, here are the llogs requested for one of the filesystems involved:
rds-d2-client.2021-09-15.llog rds-d2-OST0000.2021-09-15.llog rds-d2-MDT0000.2021-09-15.llog
I had a quick look through these and the only NIDs present in these that are not from the same LNET network that the servers are on (o2ib2), are the NIDs present in the 'nosquash_nids' parameter. This is only present in the MDT log and the client log.
Would Lustre try to get the primary NID even for these - particularly on the MDT I suppose, since that's where we observe the crash/lockup?
|
|
Hi Serguei - just an update after the last 24hrs experiences.
The issue we saw in our last call with the stuck mount hasn't re-appeared since. I don't know what that was caused by, but we've had no trouble mounting the filesystems on other clients.
Since our call, we re-added routing configuration to a number of previously stuck (non-rebooted) clients - mostly important login nodes or other such infrastructure. We didn't see any issues after doing this and as the day wore on yesterday, we started feeling more confident to add more of the OPA compute nodes (the bulk of the routed clients) back by adding routing configuration to these too.
This last action did cause another large outage as all the servers started locking up with the LNetPrimaryNID stack trace previously mentioned. As we observed in the last call, many of these clients would have had old 'peer' information at this time, that we couldn't clear with lnetctl as we observed together. I don't have any more debug traces from yesterday's crashes, so we can't verify exactly what NID the servers are trying to 'discover' here unfortunately.
Did any of the previous debug logs provided have any indication of what it's trying to look up in the LNetPrimaryNID routine?
Otherwise, since yesterday's lockup, we have rebooted all the OPA clients, so they all have cleaned up peer information now. Today we have been cautiously bringing up these OPA clients and so far we haven't seen any issues - we still have about half still to do though, so it's too early to give the all clear, but it's looking positive so far.
I think if it stays this way and we don't see any other router-related issues (such as at the start of this ticket), then our focus right now is really going to be on the LNetPrimaryNID lockup issue - what the cause of that is, and if it's seen as a bug that we could perhaps get a patch on or not? Is there anything in the debug logs provided up to now that give a clear cause of why this stack trace is being produced? Is it clear at the moment that what precipitated this is me removing an LNET network (o2ib0) from our routers - and thus from the overall routed environment?
Cheers,
Matt
|
|
Matt,
My understanding is that any unreachable NID in llog (on a remote network for which there's no route or on the local network but belonging to a node that's down) will cause this problem. I'm not sure if "nosquash_nids" are treated differently.
|
|
That was my observation too yesterday. I have seen MDS and also OSS servers locking up in this manner. I have also observed that this bug was being triggered as soon as we enabled lnet routers and clients were able to communicate with the servers. While servers had been rebooted and had clean peer information, many of the clients had peer information with NIDs that were no longer reachable. After effectively rebooting all clients and therefore cleaning old peer information on them the situation has much improved. I suspect that there is a bug that causes LNetPrimaryNID lock up and it appears that it could be triggered by a clients having unreachable NIDs in their peer information. We would like to put more effort and work with you to get to the bottom of this and find a resolution if there is no existing patch that addresses this issue.
Best regards,
Wojciech
|
|
LU-13972 addressed the problem with getting stuck indefinitely on an unreachable NID, the fix should be in 2.12.7-RC1
"Stuck" clients may also be having similar issue. You can check where they are stuck by
echo l > /proc/sysrq-trigger
(provided sysrq is enabled). Please do this on the clients and the servers that are having the issue and provide the output.
Are all of your clients/servers upgraded and using the same lustre version?
So far I don't understand how client trying to connect to an unreachable peer can cause the server to get stuck with the trace you're seeing. The debug logs I have seen don't have this captured, so if there's a way to reproduce, please try capturing another debug log.
Thanks,
Serguei.
|
|
Hi Serguei, sorry for delay in your last questions.
Yes essentially our whole environment is running 2.12.7 with a single patch on top which is from LU-13356. So yes we should have LU-13972 already in that case. There may be a small handful of clients running slightly older versions, possibly 2.12.6 or around that, but not significantly older.
Since we last updated a week ago, we haven't seen the issue return again - likely we will not any more since we have finished removing the o2ib0 network completely from the environment, including the routers, so there are no longer any peers on this network trying to talk to the servers.
On reflection it does appear to be the case that by attempting to remove the NIDs for this network from the LNET routers originally, we triggered this problem, however I still don't feel that we've quite got to the bottom of why this caused the issue, since I don't believe we had any NIDs on o2ib0 in the llog for these servers (we did post these earlier and I don't see why the llog would have changed since before we removed the o2ib0 NIDs from the routers).
Therefore unfortunately it's very unlikely that we will be able to capture another debug log of the issue at this point.
Before we close this issue - is there any thinking at your end that there is a bug in the code at this point still to be worked on, or will we not know without fully tracing the call chain in debug logs to properly see the sequence of events triggering the issue?
My main concerns at this point are:
- I'm not sure whether I did something wrong in the steps I took to shutdown the o2ib0 network originally. We have a few LNET networks in our environment, and it's going to be the case that we will be removing a network from the routers as a fabric is decommissioned from time to time. So after this experience, I feel uneasy about whether there is something we should have done to prevent us hitting this issue?
- On top of this, we had plans to have small private LNET networks added/removed from the routers more dynamically in future as part of creating small groups of clients on isolated VLANs in our environment, to which we could provide more secure, locked-down mount points via the nodemap capabilities. So a big part of this would be clients and LNET networks appearing and disappearing regularly. So again, perhaps we are missing something in the precise way that we go about removing a set of peers and associated LNET network to not trigger this problem.
What is your opinion right now? Is this a procedural error on our part, or is there a bug here to be found?
Thanks,
Matt
|
|
"[...]. We see this on all the MDS in the system and they all progressively get locked up, repeating this kind of stack trace repeatedly until they crash or are killed by pacemaker for being unresponsive.[...]"
We have seen this kind of behavior with asymmetrical routes set client but not on server: clients can communicate with servers but not the opposite (see LU-14566).
All the available thread became progressively pending discovery for the unreachable clients.
On our filesystem MR is disabled so we backport the https://review.whamcloud.com/43537 ("LU-14566 lnet: Skip discovery in LNetPrimaryNID if DD disabled").
And we activate the LNET option "drop_asym_route" to prevent this kind of situation.
|
Generated at Sat Feb 10 03:14:28 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
client.mount.debug.log
lnet-routers.png
lustre-log.1630947202.26758.gz