|
Sorry, I forgot to say an important thing regarding another symptom: when this happens, after some time, it is sometimes followed by LU-12120 (LBUG tgt_grant_incoming). I have several crash dumps of these, so maybe the answer is in the crashdumps, not sure.
|
|
The issue seems te be resolved now after we decided to mount all targets in abort_recov..., we probably lost most of the jobs on the cluster.  But the filesystem has run all night. The only remaining suspicious messages I can see now are these:
Also saw these messages shortly after bringing the filesystem back up:
I added another and more complete "foreach bt" output from a voluntary crash dump we took shortly after seeing the problem (lots of "event type -5" messages). Please see fir-io3-s2_foreach_bt_20190422_204054.txt with full logs in fir-io3-s2-vmcore-dmesg-20190422_204054.log. To me, this looks like very much another instance of LU-12162 or LU-12018 as I said originally. So perhaps the fix doesn't really work (because we have it)? bzzz what do you think?
Those backtraces in the crash dump look familiar:
Before performing the global remount in abort_recov, we tried to bring the filesystem back up many times (for several hours), but either we had these event type -5 messages + spikes in msgs_alloc etc. (= OSS deadlock for me), or a clear LBUG, like this one (looks like LU-12120):
|
|
Hmm, after looking over the logs, there's no evidence of deadlock - it looks like a network issue. Possibly the same as reported by NASA, but the signature is a bit different.
It's good to distinguish between osd_ldiskfs and ldiskfs itself, by the way. You're seeing a lot of threads stuck in osd_ldiskfs, which is a layer up from ldiskfs itself.
Having dug through the traces, etc, I don't see any evidence of deadlock. I see a node that is having an enormous amount of trouble doing network communication, and suffering because of it.
We see thousands of this message:
And these:
We're seeing a lot of evictions, apparently from the same cause. All of this chaos is affecting various worker threads, which over time have become stuck (this is your increasing load average, which reports waiting threads). They're all stuck in pretty "normal" places, waiting for I/O or communication, so if they are indeed stuck (I suspect some are truly stuck and some would make progress given time), it's because they've become confused by repeated communication failures and evictions.
At first glance, these messages are not the same as those in the NASA bug. I'll look around and see what I can find, but I'm also going to see if Amir can take a look.
|
|
Amir,
Are you able to take a look at this one? The main messages of interest seem to be the bulk callback error and the lnet_is_health_check errors I highlighted.
|
|
Hi Patrick,
Thanks SO much for having a look at this so quickly! Ok, I may have been confused with the previous deadlock that we have seen. And duly noted for osd_ldiskfs/ldiskfs, thanks!
I've looked a bit further at the logs from last night, and I see one occurrence of event type 5 (multiple messages on each OSS but nothing like before), and it happened on all OSS at the same time. But it looks like it's just a client timing out:
Also, I have looked at past OSS logs, and the event type 5 is actually present pretty often when a client timed out.
Lastly, I'm attaching two graphs, just because I wanted to show you the mess of last evening. First one is the overall OSS I/O bandwidth and the second one is the rate of msgs_alloc (all servers). The small peak around midnight is actually matching event type -5 that I've seen above.
|
|
Can you help interpret the bandwidth graph? I see positive and negative numbers but I'm not sure what they mean. Is one read and the other write?
About the msg_allocs: While it would be good for Amir to weigh in, I think those are associated with resends due to message failures.
About the -5:
Yes, -5 isn't that rare, but I think the -125 error code is? (It would be great to know if I'm wrong about that and it is in fact common.)
|
|
Oops, sorry about that, missing legend... that's bad!  . Overall read I/O from all OSS's are in blue (positive) and write in red (using negative values so it's more readable). They are taken from
/proc/fs/lustre/obdfilter/-OST/stats
Curious to know more about the server_bulk_callback errors too.
And you said there was a node "having an enormous amount of trouble doing network communication", are you able to tell me which one? or this information is missing from the logs?
Also just to add more context, we're currently completing a client upgrade to include the patch from "LU-11359 mdt: fix mdt_dom_discard_data() timeouts", so we're rebooting a lot of clients. Perhaps this was a contributing factor too...
|
|
That's possible. To clarify an earlier question:
Do you see the "-125" error code associated with those messages in earlier instances?
By the way, if you are rebooting clients, is the dip in activity perhaps from the MDS related hang on client loss we've discussed previously that you recently reminded me is still an issue? (Are you still not unmounting on reboot?  )
[Edit:]
Actually, the length of the gap mostly rules that out.
|
|
Yes, I can see many of them actually. An example is attached in fir-io1-s1-previous_bulk.log, just scroll a bit and you'll see. Then we can also see a few backtraces that NASA is seeing too. Looking at the I/O graph during that time, I see a small drop in I/O at the time of the event, and then it recovers.
|
|
Please note that we're also using the patch from LU-12065 (lnd: increase CQ entries) everywhere now (servers, routers, clients). This one was added following some networking issues on OSS that were tracked in LU-12096.
|
|
Perhaps something is wrong with our lnet config? I attached the output of lnetctl net show -v on:
We usually don't tune much lnet and try to stick to the default values.
|
|
I see quite a lot of these msgs this morning, on OSS but also on one of our MDS:
When enabling +net logging, I see some of the following messages (which are our Lustre routers):
Below, o2ib7 is Fir's IB EDR fabric,
10.0.10.5x@o2ib7 are MDS,
10.0.10.1xx@o2ib7 are OSS
10.0.10.2xx@o2ib7 are routers
Perhaps we should increase the peer_credits so that our MDS can handle more load to Sherlock's Lnet routers:
But that means we have to change it on the routers too I assume.
|
|
The lack of peer credits seems to have been a consequence of having enabled lnet debugging. We did some tests with +net at some point but this was a bad idea I guess. Setting debug and subsystem_debug back to -all seems to have fixed it. We're still seeing the following messages but it's hard to tell what could be wrong:
|
|
Stephane,
I'm kind of thinking this specific bug has been overcome by events and is maybe captured elsewhere, as we've worked through various issues recently - Are you still seeing the issues described here?
|
|
Hi Patrick,
No, not recently, at least not with these specific errors. The last problem we had on Fir is described in LU-12451. I'm not sure why we had some routers running out of rtr credits.
|
|
OK - Rather than keep open an old ticket, we'll focus in the new ones. Thanks, Stephane.
|
|
This particular issue has not reocurred recently, and there is another bug (https://jira.whamcloud.com/browse/LU-12451) capturing similar issues, so we will focus on that and re-open this if necessary.
|
Generated at Sat Feb 10 02:50:38 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
fir-io1-s1-previous_bulk.log
fir-lnet-msgs_alloc-20190423.png