|
Currently we cannot make MDT0 work again on Fir (2.12.3) due to these backtraces and lock timeout:
This looks similar to LU-11358.
I also just noticed another thing, on our new 2.12.3 clients, I can't find the async_discard import flag:
Is it unknown2_0x4000 ? Or is it missing??
Please confirm that the async_discard patch is missing from 2.12.3? If it's the case, we'll need to perform a full downgrade or emergency patching of the cluster and Lustre servers.
Attaching logs from MDS fir-md1-s1 which serves MDT0 and that we cannot make operational again at the moment. We even tried with abort_recovery with no luck.
|
|
Apparently the patch is included in 2.12.3 as
Which includes:
So why is the Lustre client version available on the Whamcloud repository not recognizing the async_discard import flag?
|
|
Note: the 2.12.3 version I build is not able to recognize the connect flag either, and shows as unknown2_0x4000. Prior 2.12.3, with our patched 2.12.0, the flag was recognized as "async_discard". If the clients don't use the new feature, that would explain the MDT deadlock after we finally upgraded 1,000+ clients.
|
|
Mike should look at this when he comes online but we will need to see what can be done to restore production in the meantime
|
|
Thanks! I've tried a abort_recovery on MDT0 (we have four MDTs), but this didn't work. I've restarted all MDTs without luck neither. MDT 1 to 3 are still working and some I/Os are still going on the cluster apparently, but the namespace is not accessible due to MDT0 being down/deadlocked. I think the next step will be to try an abort-recovery on all four MDTs but that will likely fail all jobs.
|
|
As you posted in your recent comment, 0x4000 is the unknown2_0x4000 flag, which should be interpreted as async_discard. The fact that it is being printed in the imports file shows that both the client and server understand the OBD_CONNECT2_ASYNC_DISCARD flag, though it is not being printed correctly. It appears that this is because the obd_connect_names[] array is "positional" and missing the lower-numbered flags "pcc" and "plain_layout" because those features were not backported to b2_12:
"archive_id_array",
"unknown",
"selinux_policy",
"lsom",
"async_discard",
NULL
I will make a patch so that "async_discard" is printed properly, but that doesn't seem to be the root cause of this issue.
|
|
Thanks Andreas! And previous to the async_discard patch, we were still able to put the system back online.
So I'm attaching a debug log from the MDS as fir-md1-s1-dk.log.gz . Looks like there are some lfsck errors, it's perhaps something to look at, like this one:
We started lfsck in dry-run mode this morning. This might be related.. perhaps. I know that the OI_Scrub had completed successfully on all MDTs.
Let me know what else would be helpful?
|
|
The lfsck_layout_verify_header_v1v3() message relates to LOV_PATTERN_MDT = 0x100. That is not yet implemented in (LU-11081), but should only prevent DoM files from being repaired by LFSCK.
I'm just looking at the debug log to see if there is anything I can see that might avoid this issue.
|
|
Thanks Andreas, I will follow LU-11081.
Earlier today we took a crash dump (sysrq-triggered) and I'm attaching the output of foreach bt as crash-sysrq-fir-md1-s1-foreach-bt.log. We can see many mdt threads blocked in mdt_intent_brw(). The lfsck thread is also doing things in ldiskfs but I doubt this is related to lfsck as we started to mount MDT0 using -o skip_lfsck and it was the same.
|
|
Attaching debug logs for a local client fir-rbh01 as fir-rbh01.dk.log that we restarted too, so the log is complete. Two kind of errors I noticed while gathering this:
And:
|
|
It looks like the problem is related to the file "[0x200034eb7:0x1:0x0]", which you may be able to use with "lfs fid2path" to determine what its pathname is. The lock reference count is going up and down on this file, but stays around 870 or so, likely indicating that either there is a lock refcount leak, or the clients continue getting references on the file and never actually release them.
It seems like the MDS threads are eventually completing their operations on this file:
If LFSCK is still running, you could consider to stop it (via "lctl lfsck_stop"), as that may be driving up the load on the MDS?
|
|
Thanks! We are looking into it. I've got the path after a stop/start of MDT0 but I still can't stat it or get its stripping info. The corresponding user runs a custom code (aeronautics), and has 4 jobs of 240 tasks each. Not sure if all of them are accessing this file and how big is the file at this point, but it is almost certain it is using DoM.
Re: LFSCK, it has completed and just for information I'm attaching the results as fir-md1-s1_lfsck-results.log
|
|
Andreas, the file is empty!
stat:
Stripping info:
|
|
Andreas, at this point, the filesystem seems back online again. I did several stop/start of MDT0 while I had a stat and lfs getstripe running, but it tooks several attempts to get the result. But then the file was not empty anymore and contained 186 bytes of text then after a few minutes was deleted.
MDS logs:
|
|
Updated MDS dk logs attached as fir-md1-s1-dk2.log.gz and filesystem is still running.
|
|
The file in question, DEFAULT.PKG, is actually used by the AERO-F software to restart a simulation (https://frg.bitbucket.io/aero-f/index.html#Restart)
|
|
Stephane, similar stack trace was seen several times in couple bugs like LU-11358 you've mentioned but in that cases there was also other thread which was blocker, but I don't see any other thread in logs which would block all these mdt_brw_intent() threads.
In fact DoM is vulnerable to such access patterns like we have here - multiple writers to single file, because DoM lock covers whole DoM region but not stripes, so only one writer at the time is allowed. That can become bottleneck and may cause lock timeouts on high load. So general recommendation in that case - use ordinary file with OST stripes instead.
Meanwhile that shouldn't cause such server hung as you've experienced, so I will investigate that more.
|
|
Hi Mike,
Thanks for looking at this!
You say the whole DoM region, but in our case, the DoM size is only 128 KB. It's less than the size of an OST stripe (4MB), so I'm not sure to understand why DoM wouldn't be as efficient. For larger DoM sizes, I understand.
And we do have arrays of SSDs for the MDT storage. So disk I/Os for DoM should not be a problem.
Our main problem here is that it blocked MDT0 and thus the filesystem/namespace. I was hoping you could find a blocking thread maybe from crash-sysrq-fir-md1-s1-foreach-bt.log . Do you want the associated crash dump? Let me know, I can upload it to your FTP.
|
|
A similar situation happened again after the user relaunched his jobs, but the traces on the MDS are a bit different I think, at least the first ones. Looks like the filesystem is blocked again.
One of these traces:
Attaching MDS logs as fir-md1-s1_20191105.log
|
|
Stephane, yes, I was checking exactly that sysrq file but found no good candidates. As for DoM file efficiency, it is not about its size but how many processes are accessing it at the same time. I think with file DEFAULT.PKG many processes are trying to write to the file beginning so DOM region becomes bottkeneck and OST stripes are not so useful in that case, each process need access to DOM region first each time?
|
|
Ah yes, I see, thanks. And I just found 250+ files like these that may be used by these aero-f jobs:
So they are small files but use the full 128KB DoM region but they are all on a single MDT, and accessed by 4 x 240 tasks (potentially).
I'll make changes so that these won't be using DoM anymore.
|
|
Stephane, there is one ticket for DoM improvements which can improve such access patterns, LU-10664, but it has no patch at the moment.
|
|
Thanks Mike. We're currently discussing about changing our default stripping to avoid further issues like these and perhaps only use DoM on specific cases.
FYI, yesterday, after I killed the suspected jobs, I still had to do a stop/start of MDT0 to resume filesystem operations (it didn't recover by itself).
|
|
To avoid further issues for now, we have removed the default DoM stripping from all directories on this filesystem (only kept a PFL stripping). New files won't use DoM anymore. We'll see if that helps.
|
|
Stephane, it would make sense to get an strace (or equivalent Lustre "{{lctl set_param debug=+vfstrace +dlmtrace) from these jobs to see just how many times they write to the same file.
Mike, since it is possible to migrate DoM components to OSTs (either with full-file copy in 2.12 or via FLR mirror in 2.13 patch https://review.whamcloud.com/35359 "LU-11421 dom: manual OST-to-DOM migration via mirroring"), have you thought about automatically migrating files with high write lock contention from DoM to a regular OST object? Since the amount of data to be moved is very small (under 150KB in this case), the migration should be very fast, and it would allow extent locks to be used on the file.
That said, I have no idea how hard this would be, and only makes sense if there are multiple writers repeatedly contending on the same DoM file component (which I suspect is rare in most cases). Even here, it may be that if the clients are only writing to the same file a handful of times that the extra migration step would make the performance worse rather than better. If they write to the same file hundreds of times then it might be worthwhile to implement.
Even in IO-500 ior-hard-write the chunk size is 47008 bytes, so at most 2-3 ranks would be contending on a 64KB or 128KB DoM component, and we never had problems with this in our testing.
|
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36881
Subject: LU-12935 obdclass: fix import connect flag printing
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: b2afbb33175af02608d9d4127370dc874542148b
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36881/
Subject: LU-12935 obdclass: fix import connect flag printing
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: b66efa7d500f94d755f318d43804f0f3eb883835
|
|
Landed for 2.12.4 - does not affect master
|
Generated at Sat Feb 10 02:56:56 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
crash-sysrq-fir-md1-s1-foreach-bt.log
fir-md1-s1-dk.log.gz