|
Lai
Could you please investigate?
Thanks
Peter
|
|
Another thing, I've been told by the cluster sysadmin that this problem often starts just after creating a batch of new users / groups in the morning, so some mkdir in /fir/users or /fir/groups hang, this is just a hint... but this would mean it is a DNEv2-specific issue indeed.
|
|
We got a new occurrence of the issue last night, hung filesystem on several clients, including 3 out of 8 login nodes, okay on the others. Super weird but very impacting issue. I gathered more logs first:
I'm attached the corresponding .gz files to this ticket.
Same kinds of call traces again:
not sure it is related, but just in case, ldlm lock count when this happened:
Then tried lru_size=clear on all nodes without success (filesystem was still hung), however the lock counters dropped a bit:
Finally I had to restart all MDTs for the filesystem to come back online.
|
|
I'm also wondering if the garbage found in the lustre dk logs are normal:
|
|
Stephane, when you are creating user/group directories, is it just a simple "mkdir" at the top level, or is there a script and/or command that is run to create those directories? It looks like all of your stack traces involve remote renames, which definitely makes things more complex on the DNE front than a simple mkdir. Is it possible to capture a debug log of a (sample) user/group creation on the client, so we can see what operations are being done there?
|
|
Hmm ok. We don't think that it is related to the account creation process anymore as no new accounts were created during last weekend (last one was on Friday), but the issue happened last night anyway.
Account creation is basically this:
But due to our DNE setup (each user (or group) directory is randomly stripped to a single MDT) , if the issue is due to remote renames, that would mean mv between different user or group directories, that is not impossible, but I would guess it's rather unlikely. Although, one possible case would be a user that moves data to/from his $SCRATCH (/fir/users/<user>) to/from $GROUP_SCRATCH (/fir/groups/<group>), as both directories are not necessarily on the same MDT in that case.
Do you see any interesting FIDs that I could check?
|
|
It's happening again, and this time even the recovery failed 
|
|
Finally, it looks like all MDTs restart completed successfully:
But this issue is really annoying. We were creating home dirs when this happened and mkdir was blocked. However, we're creating new users all the time so I'm not sure this is really realted. To do something at least, we're gonna get rid of the DNEv2 stuff for now, and do a lfs mkdir to create new user dirs, to see if that helps.
|
|
Stephane, can you dump all process backtraces on all MDS's by 'echo t > /proc/sysrq-trigger' next time? Which may help identify there are real deadlocks in MDS, or maybe it's just caused by some slow clients not canceling locks, and MDT has to wait for client eviction and recovery finish, and renames between MDTs causes more lock dependencies and finally cause the system looks like deadlock.
I'm checking LDLM code to see if we can fail MDT local lock request after timeout instead of wait indefinitely to avoid deep lock dependencies, I'll update you if this is working.
|
|
Thanks Lai, will do!
|
|
Hello,
I wanted to update this ticket because I just discovered that I provided some wrong information before.
We were actually using rename during the account creation process, basically something like:
cd /fir/users
mkdir $tmpdir
mv $tmpdir $user_scratch
chown $user: $user_scratch
chmod 700 $user_scratch
lfs setquota [...]
Because /fir/users is a DNEv2 directory, that probably explains why we see some remote renames in the backtrace then.
For now, we have switched to lfs mkdir to avoid DNEv2 (and with no rename) in order to stabilize the filesystem (because it has been painful). Perhaps next week at some point, we can try again to do mkdir + rename to see if I can get MDS traces.
|
|
Using lfs mkdir will still invoke DNE1, which means remote entries for many of the directories, because you won't know in advance what MDT the directory will hash to. Using simple "mkdir" will not invoke DNE because it will only create the name and directory on the same MDT, so long as you don't use rename in the process.
|
|
Thanks Andreas, I think I get it!  We'll just use mkdir without rename for now.
|
|
This happened again a few minutes ago (without us creating any new account today), but users can always move data from different group spaces so...
I had to restart all MDTs to fix the issue. This time, the recovery was fast. Also, I was also able to dump the tasks using sysrq on both MDS when this happened.
Attaching fir-md1-s1_sysrq-t_20190308.log.gz (MDT0000 and MDT0002) and fir-md1-s2_sysrq-t_20190308.log.gz (MDT0001 and MDT0003). Hope that helps...
|
|
This issue hit us again and we were mostly down last night. The strange thing is that a few compute nodes could still access the filesystem, so it was not a total deadlock. Anyway, I dumped the tasks and also took a crash dump of both MDS.
I have gathered the following files to help troubleshoot this issue:
Attached to this ticket:
- fir-md1-s1_console_full_20190309.log.gz - full console logs of fir-md1-s1 (MDT0000 and MDT0002) from the last MDT startup (Fri Mar 8 16:34:33 2019), including the dump of the tasks with SysRq, the reboot (power diag) and the restart of the MDTs with the recovery.
- fir-md1-s2_console_full_20190309.log.gz - same for fir-md1-s2 (MDT0001 and MDT0003)
Uploaded to the FTP server, two tarballs of the MDS crash dump:
- fir-md1-s1-crash-2019-03-09-10-11-07.tar.gz
- fir-md1-s2-crash-2019-03-09-10-14-21.tar.gz
Kernel version is 3.10.0-957.1.3.el7_lustre.x86_64 and I believe the debuginfo rpms have already been uploaded (kernel-debuginfo-3.10.0-957.1.3.el7_lustre.x86_64). Let me know if anything else is needed.
We really appreciate your help in this matter!
|
|
Same issue happened 4 or 5 times today, each time restarting the MDTs unblocked the situation, but sometimes the problem comes just after a few minutes. This is greatly affecting our users and their research work and I don't know what to do to avoid it at this point... I hope you'll be able to provide some workaround or patch. Thanks!
|
|
Stephane, is this always still happening with rename between MDTs? Is it possible that your DNE2 striping is inherited into the subdirectories so that any rename is a cross-MDT rename?
One option to avoid the DNE deadlocks is to return -EXDEV for cross-MDT renames instead of doing the rename, which causes mv to copy the file in userspace. It would be possible to make a patch to allow this behavior to be configured.
There is a single DLM lock that is protecting renames in the filesystem to avoid circular directory trees. At one point we discussed allowing renames of regular files within a directory to happen in parallel with only a local lock so that there isn't so much contention on the global rename lock. However, I'm not sure if that is implemented yet.
|
|
Hi Andreas! I don't think our DNE2 stripping is inherited into the subdirectories, unless I missed something (more below). Each time, I can see backtraces involving mdt_reint_rename* and mdt_remote_object_lock*. But it's strange that they are so many renames, they should be pretty rare in our setup indeed.
Typical trace:
I checked every subdirectory in each user directory and I didn't find any inconsistency:
They all report 'none' as the lmv_hash_type, so I guess that means DNE1:
Typical subdirs:
It could be that a user is doing renames from $GROUP_SCRATCH to $SCRATCH (from/to a deep subdirectory), but otherwise I don't see why there would be so many remote renames.
Would there be a way to enable traces on the MDT so I could find the FIDs involved in remote renames? That way I may be able to spot a specific user or job.
And yes, having the possibility to enable -EXDEV on remote rename would be great.
|
|
There are mdt.*.rename_stats files for all the MDTs that can tell you the number of renames, both within a directory and across directories. You can also check mdt.*.exports.*.stats and mdt.*.job_stats to see which clients or jobs are doing a large number of rename operations. It may well be that some new workload is doing a lot of renames.
|
|
Perhaps some specific workload is doing this yes, because for example today, things are running okay so far...
rename_stats shows quite a lot of crossdir renames:
[root@fir-md1-s1 ~]# lctl get_param mdt.*.rename_stats
mdt.fir-MDT0000.rename_stats=
rename_stats:
- snapshot_time: 1552344763.503420069
- same_dir
4KB: { sample: 124757, pct: 65, cum_pct: 65 }
8KB: { sample: 0, pct: 0, cum_pct: 65 }
16KB: { sample: 31900, pct: 16, cum_pct: 82 }
32KB: { sample: 1594, pct: 0, cum_pct: 83 }
64KB: { sample: 2436, pct: 1, cum_pct: 84 }
128KB: { sample: 3635, pct: 1, cum_pct: 86 }
256KB: { sample: 1374, pct: 0, cum_pct: 87 }
512KB: { sample: 4473, pct: 2, cum_pct: 89 }
1MB: { sample: 20189, pct: 10, cum_pct: 100 }
- crossdir_src
4KB: { sample: 88, pct: 24, cum_pct: 24 }
8KB: { sample: 0, pct: 0, cum_pct: 24 }
16KB: { sample: 158, pct: 43, cum_pct: 67 }
32KB: { sample: 69, pct: 18, cum_pct: 86 }
64KB: { sample: 35, pct: 9, cum_pct: 95 }
128KB: { sample: 0, pct: 0, cum_pct: 95 }
256KB: { sample: 15, pct: 4, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 148, pct: 40, cum_pct: 40 }
8KB: { sample: 0, pct: 0, cum_pct: 40 }
16KB: { sample: 139, pct: 38, cum_pct: 78 }
32KB: { sample: 44, pct: 12, cum_pct: 90 }
64KB: { sample: 34, pct: 9, cum_pct: 100 }
mdt.fir-MDT0002.rename_stats=
rename_stats:
- snapshot_time: 1552344763.503679596
- same_dir
4KB: { sample: 28442, pct: 14, cum_pct: 14 }
8KB: { sample: 0, pct: 0, cum_pct: 14 }
16KB: { sample: 18463, pct: 9, cum_pct: 23 }
32KB: { sample: 116725, pct: 59, cum_pct: 83 }
64KB: { sample: 2574, pct: 1, cum_pct: 84 }
128KB: { sample: 5828, pct: 2, cum_pct: 87 }
256KB: { sample: 23572, pct: 12, cum_pct: 99 }
512KB: { sample: 14, pct: 0, cum_pct: 99 }
1MB: { sample: 7, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 56, pct: 0, cum_pct: 0 }
8KB: { sample: 0, pct: 0, cum_pct: 0 }
16KB: { sample: 136, pct: 0, cum_pct: 0 }
32KB: { sample: 4, pct: 0, cum_pct: 0 }
64KB: { sample: 0, pct: 0, cum_pct: 0 }
128KB: { sample: 3969, pct: 12, cum_pct: 12 }
256KB: { sample: 7792, pct: 23, cum_pct: 36 }
512KB: { sample: 11381, pct: 34, cum_pct: 70 }
1MB: { sample: 9710, pct: 29, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 1607, pct: 4, cum_pct: 4 }
8KB: { sample: 0, pct: 0, cum_pct: 4 }
16KB: { sample: 1703, pct: 5, cum_pct: 10 }
32KB: { sample: 4349, pct: 13, cum_pct: 23 }
64KB: { sample: 6850, pct: 20, cum_pct: 43 }
128KB: { sample: 8948, pct: 27, cum_pct: 70 }
256KB: { sample: 4643, pct: 14, cum_pct: 85 }
512KB: { sample: 4946, pct: 14, cum_pct: 99 }
1MB: { sample: 2, pct: 0, cum_pct: 100 }
mdt.fir-MDT0001.rename_stats=
rename_stats:
- snapshot_time: 1552345057.683942486
- same_dir
4KB: { sample: 285321, pct: 98, cum_pct: 98 }
8KB: { sample: 0, pct: 0, cum_pct: 98 }
16KB: { sample: 639, pct: 0, cum_pct: 98 }
32KB: { sample: 331, pct: 0, cum_pct: 98 }
64KB: { sample: 1918, pct: 0, cum_pct: 99 }
128KB: { sample: 180, pct: 0, cum_pct: 99 }
256KB: { sample: 26, pct: 0, cum_pct: 99 }
512KB: { sample: 0, pct: 0, cum_pct: 99 }
1MB: { sample: 0, pct: 0, cum_pct: 99 }
2MB: { sample: 1623, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 182, pct: 1, cum_pct: 1 }
8KB: { sample: 0, pct: 0, cum_pct: 1 }
16KB: { sample: 0, pct: 0, cum_pct: 1 }
32KB: { sample: 0, pct: 0, cum_pct: 1 }
64KB: { sample: 1194, pct: 6, cum_pct: 7 }
128KB: { sample: 0, pct: 0, cum_pct: 7 }
256KB: { sample: 0, pct: 0, cum_pct: 7 }
512KB: { sample: 0, pct: 0, cum_pct: 7 }
1MB: { sample: 0, pct: 0, cum_pct: 7 }
2MB: { sample: 0, pct: 0, cum_pct: 7 }
4MB: { sample: 0, pct: 0, cum_pct: 7 }
8MB: { sample: 16000, pct: 92, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 588, pct: 3, cum_pct: 3 }
8KB: { sample: 0, pct: 0, cum_pct: 3 }
16KB: { sample: 297, pct: 1, cum_pct: 5 }
32KB: { sample: 485, pct: 2, cum_pct: 7 }
64KB: { sample: 6, pct: 0, cum_pct: 7 }
128KB: { sample: 2104, pct: 12, cum_pct: 20 }
256KB: { sample: 2181, pct: 12, cum_pct: 32 }
512KB: { sample: 4139, pct: 23, cum_pct: 56 }
1MB: { sample: 7576, pct: 43, cum_pct: 100 }
mdt.fir-MDT0003.rename_stats=
rename_stats:
- snapshot_time: 1552345057.684301349
- same_dir
4KB: { sample: 392450, pct: 90, cum_pct: 90 }
8KB: { sample: 0, pct: 0, cum_pct: 90 }
16KB: { sample: 11513, pct: 2, cum_pct: 93 }
32KB: { sample: 6056, pct: 1, cum_pct: 95 }
64KB: { sample: 17452, pct: 4, cum_pct: 99 }
128KB: { sample: 2634, pct: 0, cum_pct: 99 }
256KB: { sample: 1281, pct: 0, cum_pct: 99 }
512KB: { sample: 1, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 188, pct: 10, cum_pct: 10 }
8KB: { sample: 0, pct: 0, cum_pct: 10 }
16KB: { sample: 0, pct: 0, cum_pct: 10 }
32KB: { sample: 1656, pct: 89, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 630, pct: 34, cum_pct: 34 }
8KB: { sample: 0, pct: 0, cum_pct: 34 }
16KB: { sample: 530, pct: 28, cum_pct: 62 }
32KB: { sample: 684, pct: 37, cum_pct: 100 }
But what does '16,000 x 8MB crossdir_src' mean, for example?
I'm going to track the stats per job, thanks for the guidance!
|
|
The "samedirr" stats count the number of renames vs. the size of the directory for local renames, while "crossdir_src" and "crossdir_tgt" count the number of renames for a given the source and target directory size. This allows seeing whether renames are typically in large or small directories, and whether they are typically within a single directory or between directories.
It looks like the 16000 renames from the 8MB directory (which is fairly large) are to smaller target directories, or possibly the same target that is growing larger as it is used? Maybe someone moving a bunch of files around?
I see that these stats do not show if the rename is from a local or remote MDT, which would be useful.
|
|
Ok. thanks!
So the issue just occurred again, so I checked exports.*.stats at that time and there were only 4 nids showing crossdir_rename operations:
The last one, 10.8.0.66@o2ib6, is a login node (sh-ln06), so that makes sense, people usually use them to organize/move their files around. The others are compute nodes. But overall, it doesn't seem there are a lot of occurrences of crossdir_rename. I'll track these more closely, now that the MDTs have been restarted.
|
|
fir-md1-s2 (serving MDT0001 and MDT0003) has started to show the typical backtraces:
I am attaching a dump of the lock table of this MDS as fir-md1-s2-dlmtrace_20190311.log.gz, perhaps that could help. I'll do it again when the filesystem is really hung. No backtrace can be found yet on fir-md1-s1 (MDT0000 and MDT0002) at this time.
|
|
Sorry, I just realized that the backtraces are not the typical ones, so it might be another issue.. in any case, the filesystem is still responsive at the moment, so perhaps just ignore my last comment!
|
|
Hi Stephane, the backtrace shows one rename thread is stuck in mdt_dom_discard_data():
Which causes all subsequent renames waiting on rename_lock, I'll commit a fix soon.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34401
Subject: LU-12037 mdt: call mdt_dom_discard_data() after rename unlock
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 521781819bc780f201445c1385eed49051756342
|
|
Lai, this can be fixed by https://review.whamcloud.com/#/c/34071/ patch already, if you have a reproducer, can you check that?
|
|
Hi Mike, yes, your patch can fully fix mdt_dom_discard_data() deadlock, but it may take a while to pass review and land, and this fix is trivial, which can be used now.
Stephane, you can update MDS with https://review.whamcloud.com/34401 and see whether such deadlock still happens.
|
|
Awesome! Thanks! I just patched both MDS with Lai's patch. We'll see how it goes, I'll keep you posted!
|
|
Lai, can you also work on a separate patch that adds a new mdt.*.enable_remote_rename parameter (default = 1, but can be disabled if needed) that replaces remote renames with an -EXDEV return value, so that the client will do a userspace copy of the files. This should be relatively straight forward I think.
We may not need this if your patch is successful, but it may still be useful if there are DNE rename problems even after the patch, or as a workaround for other problems in the future.
|
|
Andreas, okay, I'll work on it later.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34410
Subject: LU-12037 mdt: add option for cross-MDT rename
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a80e3621b65d93f10510fed8b5916d3387c5dc87
|
|
Unfortunately, https://review.whamcloud.com/34401 doesn't seem to fix our issue, while the filesystem has been working fine (even very well) for ~36 hours, then the issue occurred again at 2:45AM and then the filesystem was not available for the rest of the night and users have already reported I/O issues from jobs and login nodes.
It seems that the first error message was:
Then also MDT0000 and MDT0002 lost connectivity (?? there are both running on fir-md1-s1):
Then the usual call trace:
We restarted all MDTs this morning at 7:58. After a few minutes, all I/Os have resumed. I'm attaching full kernel logs for both MDS servers (fir-md1-s1_kern_20190314.log.gz and fir-md1-s2_kern_20190314.log.gz). Note that this was with Lustre build version 2.12.0_lu12037 (at the end of the log files).
|
|
It happened again just now.
This time I was able to dump the ldlm lock tables on both MDS while the filesystem was stuck. Attaching files fir-md1-s1_dkdlmtrace_20190314.log.gz and fir-md1-s2_dkdlmtrace_20190314.log.gz
|
|
The only "recent" mdt_dom_discard_data is this one:
|
|
Did you try the patch https://review.whamcloud.com/34410 yet?
|
|
Not yet Andreas, we'll use it in last resort. Last night has been ok so perhaps Lai's patch is helping having less frequent deadlocks after all (or the workload has changed, I don't know). What about https://review.whamcloud.com/#/c/34071/ ? Would it be interesting for you if we try this one first, or is that too risky (too big a change)? Thx!
|
|
Stephane,
Mike and I talked over that patch this morning and we think you should give it a try. It is a client & server patch, just FYI, not server only (Just pointing that out because 34410 is server only).
|
|
Hi Patrick,
Ok, we're willing to try it then if a rolling upgrade of the clients is possible. So can I patch the 2xMDS now and then the clients progressively over say 1-2 weeks?
Also just to confirm, OSS don't need the patch right?
Thanks!!
|
|
It should be possible - There is a small chance that an un-upgraded client could get an error when writing to a file that is being deleted (final unlink). But that should be rare, and since you are unlinking the file, the error does not represent a real problem.
And yes, not needed on the OSS. Just MDS and clients.
|
|
Stephane, I've just checked that - it is better to upgrade clients along with servers, clients may experience assertion with that patch, that is why client changes were made in fact. It seems that client can be safely updated also prior server. Let me check that and also think if that is possible upgrade server first, maybe with patch modification
|
|
Thanks much! I haven't updated anything yet, so just let me know.
|
|
Filesystem blocked again, I'll need to know ASAP if we it's doable and in what order we can apply your patch Mike, otherwise I might have to fail back to the EXDEV patch. Interestingly this time I don't see any mdt_dom_discard_data trace, and the other traces are not only remote renames.. I hope it's not yet another issue with 2.12...
|
|
I guess one comment that is worthwhile to make in this ticket is that it seems your system is using PFL and DoM in addition to DNE. I missed that initially, since it wasn't actually written anywhere in this ticket, but the reference to LU-11964 and the problems with mdt_dom_discard_data() that this is the case.
Note that the https://review.whamcloud.com/34410 -EXDEV patch just adds a tunable that can turn on/off cross-MDT renames, so it should be relatively safe to apply this patch and have it available, even if you don't immediately intend to use it. To disable remote renames you would run "lctl set_param mdt.*.enable_remote_rename=0" on the MDS nodes (default is enable_remote_rename=1), which essentially reverts the MDS rename behaviour to DNE1 when remote directory rename was not implemented so that cross-MDT renames are handled in userspace like a cross-filesystem rename.
The main drawback is that it could increase data traffic significantly if they are moving large files between /fir/users and /fir/groups that are on different MDTs on a regular basis, but maybe that is acceptable in the short term to avoid MDS hangs until this issue is resolved.
|
|
Yes sorry, we're definitively looking for trouble by using PFL, DNE and DoM (and ldiskfs backend, not ZFS). Our goal when designing this system was to improve mixed workload performance and ideally read I/O performance with small files (eg. AI training sets with million of files...), without impacting traditional HPC workloads.
On a positive note, we've seen excellent performance results with these features so far on 2.12, especially PFL and read performance with large files (our filesystem was able to reach 48GB/s of sustained read bandwidth recently with jobs using many large wide-stripped files of 300-400GB each). DNE has been helping us to scan the filesystem faster (up to 16,000 files/sec with Robinhood with 16 threads) and DoM's advantage is more difficult to evaluate but it is probably helping on multiple fronts (likely one of the main advantage is to reduce the seeks on the OSTs - which are spinning disk based - for small files), but I also know that in our case, DOM makes a Linux kernel compilation on Lustre a bit faster (when the filesystem is not loaded).
Anyway, I digress, as our main issue now is to fix this MDT deadlock. So, you're right.. and I have prepared Lai's patch that adds enable_remote_rename (https://review.whamcloud.com/34410 ). This patch seems to depends on Lai's previous patch (https://review.whamcloud.com/34401) that I added too, but we're already using it. The -EXDEV patch doesn't apply cleanly but I was able to fix it for 2.12 easily. It looks like a mdt_readonly flag was added to master that doesn't exist in 2.12. So I'm ready to switch to this new patched version on our 2 MDS, either at the next occurrence of the issue, or maybe later tonight. I'll keep you posted. Thanks a lot (all of you) for the help as always...
|
|
Stephane, for now the only way to apply patch https://review.whamcloud.com/#/c/34071/ is 'client first' approach. I am still thinking how to resolve that in 'the server first' way, anyway that will be needed for compatibility but I have no solution right now.
As for other stack traces you seen - I think I saw them also, rename stack and getattr stacks, so that can be different issue with rename.
|
|
Ok thanks Mike! For now, in order to buy more time and avoid (I hope!) the MDS hangs, I have just switched off remote rename. Lai's patch seems to work as expected:
1. remote_rename switched off on all 4 MDTs:
2. On the cluster, example with mv between two dirs that not on the same MDT:
|
|
Stephane, please hold on with patch https://review.whamcloud.com/#/c/34071/, I am concerning about one thing with it and have to check that first.
|
|
Ok Mike. I won't use it, no worries. So far it's working without major issue (with -EXDEV). The only backtrace I noticed on a MDS was this one last night:
But I don't think it had an impact on the whole filesystem, we have no report of issue at this point. If this is confirmed that DOM+DNE2 on 2.12 is usable in production this way, it would be nice to have both Lai's patches included in 2.12.1.
|
|
I'd consider the -EXDEV patch more of a workaround than a long-term solution. For large files this can have a significant overhead in terms of data copies. I think anything we could do with -EXDEV to punt data copies to userspace (eg. heuristics to copy small files between MDTs instead of creating remote entries) could also be done inside lustre.
|
|
Ok, but stability is paramount at this point to restore our users' trust. I really hope Mike will find out what's wrong with the current implementation, because I agree that cross-MDTs rename is a great feature. Thanks!
|
|
Definitely yes, getting back to a stable working environment is critical. I'm not suggesting that the -EXDEV patch shouldn't land, since it is configurable, I just wanted it to be clear that landing that patch is not the conclusion of this issue.
|
|
Yep, I'm totally in line with you Andreas.
Additionally, I wanted to report more issues last night, I don't know if it's directly related so I opened LU-12091 for that in order to provide more details. It seemed to have impacted only a few clients.
|
|
After about a week, our two 2.12 MDS seem stable when using the two patches from Lai including -EXDEV and enable_remote_rename=0. Not ideal, but stable! Fingers crossed!
|
|
ARGH! I spoke too soon.... filesystem was completely stuck about 30 minutes ago, and I found a lot of call traces involving {{mdt_reint_rename }}on both MDS.
As a reminder, we are running with the -EXDEV patch and have set enable_remote_rename=0 on all MDTs:
So I was able (I hope) to dump the ldlm locks on both servers while the filesystem was stuck:
- fir-md1-s1_dkdlmtrace_20190327.log.gz
- fir-md1-s2_dkdlmtrace_20190327.log.gz
And kernel logs are also attached:
- fir-md1-s1-kern_20190327.log.gz
- fir-md1-s2-kern_20190327.log.gz
A typical call trace is:
|
|
It looks like there may still be some improvements needed for the patch https://review.whamcloud.com/34410 "LU-12037 mdt: add option for cross-MDT rename", since it is still getting the "Big Filesystem Lock" (BFL) on MDT0000 for every rename. The -EXDEV return could be done before any locking is done to reduce contention on the MDS and avoid whatever deadlock (or overload and timeout) that is being seen in that code.
Separately, it should be possible to completely avoid the use of BFL for renaming regular files within a single directory on the same MDS, and between different directories on the same MDS by locking only the directory itself. According to the rename_stats data previously posted in comment-243700, there were 1107408 same-directory renames vs. only 52629 cross-directory renames (4.75%). While this is not conclusive in terms of local vs. remote MDT or file vs. directory, it is likely that most same-directory renames are regular files on the same MDT, unless striped directories are widely used, so we could avoid the BFL for over 95% of renames, and possibly more. This improvement is being tracked via LU-12125.
|
|
Stephane, I updated patch https://review.whamcloud.com/34410, and next time when you see deadlock, it's better to dump process backtraces on all MDS's to help identify which process is the root of the deadlock.
|
|
Thanks much to both of you! I'm glad that you were able to identify the problem there.
Andreas, striped directories are definitively not widely used in our case (and we plan to keep it as is). And I'll follow LU-12125, sounds interesting!
Lai, ok I'll make sure to dump all tasks with sysrq next time. I'll keep an eye at the patch and update our MDS once it has positive reviews.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34401/
Subject: LU-12037 mdt: call mdt_dom_discard_data() after rename unlock
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 480a221aa7e76f001288c8b1127e14e7922ce497
|
|
We've upgraded the two MDS of Fir with Lai's latest patch 5 from
https://review.whamcloud.com/#/c/34410/
and we set mdt.*.enable_remote_rename=0
However, after a few hours of production, I'm still seeing a few crossdir renames in the stats on MDT0000 and MDT0002.
Just FYI, on Fir's MDS, we're now running 2.12.0 with the following patches:
LU-12065 lnd: increase CQ entriesLU-12037 mdt: add option for cross-MDT rename (patchset 5 ported to 2.12.0)LU-12037 mdt: call mdt_dom_discard_data() after rename unlock (the one that just landed in master)
and the following config changes:
- osc.*.idle_timeout=0 (
LU-11976)
- mdt.*.enable_remote_rename=0 (-EXDEV)
and a workaround to minimize LU-11989:
|
|
Note that patch https://review.whamcloud.com/34410 "LU-12037 mdt: add option for cross-MDT rename" is related to disabling renames to a remote MDT. It does not disable renames between directories on the same MDT (which would typically be the same for the per-user or per-group directories). Even in a striped directory, some fraction of renames may be on the same MDT.
|
|
Ok Andreas, thanks!
We just got a new occurrence of a hung filesystem, even with the patches... I was able to dump the kernel tasks and also the lustre ldlm locks using the commands that Patrick provided before, while everything was hung.
Looks like the issue started at around 15:45 today.
There is a first backtrace in mdt_dom_discard_data, then others with mdt_remote_object_lock:
File attached:
- fir-md1-s1-console-20190403.gz full fs lifecycle console logs of MDS with MDT0000 and MDT0002
- fir-md1-s2-console-20190403.gz full fs lifecycle console logs of MDS with MDT0001 and MDT0003
- fir-md1-s1_20190403-dk.log.gz ldlm.dump_namespaces for fir-md1-s1
- fir-md1-s2_20190403-dk.log.gz ldlm.dump_namespaces for fir-md1-s2
|
|
Perhaps an issue with unlink on rename() - which should be legit.
mdt_dom_discard_data is called by mdt_reint_rename in that case:
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34615
Subject: LU-12037 mdt: call mdt_dom_discard_data() after rename unlock
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: e6a9a9ed6bc1dc4ee92fdf75c03da2dae45eeab3
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34616
Subject: LU-12037 mdt: add option for cross-MDT rename
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 4f9b5eead1616db8cb1878194b6c2131b8a2aa65
|
|
Issue occurring again even with the EXDEV patch:
[root@fir-md1-s2 ~]# lctl get_param ldlm.namespaces.mdt-fir-MDT000*_UUID.lock_count
ldlm.namespaces.mdt-fir-MDT0001_UUID.lock_count=2813920
ldlm.namespaces.mdt-fir-MDT0003_UUID.lock_count=1110534
[root@fir-md1-s2 ~]# lctl get_param mdt.*.rename_stats
mdt.fir-MDT0001.rename_stats=
rename_stats:
- snapshot_time: 1554747505.901862704
- same_dir
4KB: { sample: 440634, pct: 89, cum_pct: 89 }
8KB: { sample: 0, pct: 0, cum_pct: 89 }
16KB: { sample: 7257, pct: 1, cum_pct: 91 }
32KB: { sample: 5923, pct: 1, cum_pct: 92 }
64KB: { sample: 32738, pct: 6, cum_pct: 99 }
128KB: { sample: 3016, pct: 0, cum_pct: 99 }
256KB: { sample: 1779, pct: 0, cum_pct: 99 }
512KB: { sample: 0, pct: 0, cum_pct: 99 }
1MB: { sample: 0, pct: 0, cum_pct: 99 }
2MB: { sample: 0, pct: 0, cum_pct: 99 }
4MB: { sample: 2, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 609, pct: 1, cum_pct: 1 }
8KB: { sample: 0, pct: 0, cum_pct: 1 }
16KB: { sample: 2925, pct: 6, cum_pct: 8 }
32KB: { sample: 11090, pct: 25, cum_pct: 33 }
64KB: { sample: 25807, pct: 59, cum_pct: 93 }
128KB: { sample: 0, pct: 0, cum_pct: 93 }
256KB: { sample: 3000, pct: 6, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 11631, pct: 26, cum_pct: 26 }
8KB: { sample: 0, pct: 0, cum_pct: 26 }
16KB: { sample: 11490, pct: 26, cum_pct: 53 }
32KB: { sample: 18106, pct: 41, cum_pct: 94 }
64KB: { sample: 1587, pct: 3, cum_pct: 98 }
128KB: { sample: 617, pct: 1, cum_pct: 100 }
mdt.fir-MDT0003.rename_stats=
rename_stats:
- snapshot_time: 1554747505.902158388
- same_dir
4KB: { sample: 69919, pct: 28, cum_pct: 28 }
8KB: { sample: 0, pct: 0, cum_pct: 28 }
16KB: { sample: 19406, pct: 7, cum_pct: 36 }
32KB: { sample: 27998, pct: 11, cum_pct: 48 }
64KB: { sample: 27143, pct: 11, cum_pct: 59 }
128KB: { sample: 9504, pct: 3, cum_pct: 63 }
256KB: { sample: 6856, pct: 2, cum_pct: 66 }
512KB: { sample: 5559, pct: 2, cum_pct: 68 }
1MB: { sample: 7179, pct: 2, cum_pct: 71 }
2MB: { sample: 14867, pct: 6, cum_pct: 77 }
4MB: { sample: 31429, pct: 12, cum_pct: 90 }
8MB: { sample: 22948, pct: 9, cum_pct: 100 }
- crossdir_src
4KB: { sample: 193, pct: 2, cum_pct: 2 }
8KB: { sample: 0, pct: 0, cum_pct: 2 }
16KB: { sample: 0, pct: 0, cum_pct: 2 }
32KB: { sample: 1877, pct: 25, cum_pct: 28 }
64KB: { sample: 4301, pct: 58, cum_pct: 86 }
128KB: { sample: 0, pct: 0, cum_pct: 86 }
256KB: { sample: 986, pct: 13, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 976, pct: 13, cum_pct: 13 }
8KB: { sample: 0, pct: 0, cum_pct: 13 }
16KB: { sample: 985, pct: 13, cum_pct: 26 }
32KB: { sample: 2449, pct: 33, cum_pct: 59 }
64KB: { sample: 2449, pct: 33, cum_pct: 93 }
128KB: { sample: 498, pct: 6, cum_pct: 100 }
I'll restart the MDTs
|
|
Patrick,
We just applied https://review.whamcloud.com/#/c/34410/6
on both of our MDS on Fir (note for self: 2.12.0.pl4)
Will need some time to confirm it's ok I think.
|
|
Thanks Stephane - Please keep us updated, particularly if you still see any cross mdt renames...
|
|
Doesn't seem to work and on top of that, we got another MDS deadlock (I opened LU-12178).
Stats for MDT0000 and MDT0002 (they were on the MDS that didn't crash):
mdt.fir-MDT0000.rename_stats=
rename_stats:
- snapshot_time: 1554957753.619746357
- same_dir
4KB: { sample: 101272, pct: 77, cum_pct: 77 }
8KB: { sample: 0, pct: 0, cum_pct: 77 }
16KB: { sample: 3282, pct: 2, cum_pct: 80 }
32KB: { sample: 7074, pct: 5, cum_pct: 85 }
64KB: { sample: 6166, pct: 4, cum_pct: 90 }
128KB: { sample: 11437, pct: 8, cum_pct: 99 }
256KB: { sample: 860, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 283, pct: 56, cum_pct: 56 }
8KB: { sample: 0, pct: 0, cum_pct: 56 }
16KB: { sample: 153, pct: 30, cum_pct: 87 }
32KB: { sample: 62, pct: 12, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 454, pct: 91, cum_pct: 91 }
8KB: { sample: 0, pct: 0, cum_pct: 91 }
16KB: { sample: 0, pct: 0, cum_pct: 91 }
32KB: { sample: 44, pct: 8, cum_pct: 100 }
mdt.fir-MDT0001.rename_stats=
rename_stats:
- snapshot_time: 1554957753.619960398
- same_dir
4KB: { sample: 309, pct: 38, cum_pct: 38 }
8KB: { sample: 0, pct: 0, cum_pct: 38 }
16KB: { sample: 41, pct: 5, cum_pct: 43 }
32KB: { sample: 41, pct: 5, cum_pct: 48 }
64KB: { sample: 414, pct: 51, cum_pct: 100 }
- crossdir_src
4KB: { sample: 2, pct: 0, cum_pct: 0 }
8KB: { sample: 0, pct: 0, cum_pct: 0 }
16KB: { sample: 0, pct: 0, cum_pct: 0 }
32KB: { sample: 421, pct: 99, cum_pct: 99 }
64KB: { sample: 1, pct: 0, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 104, pct: 24, cum_pct: 24 }
8KB: { sample: 0, pct: 0, cum_pct: 24 }
16KB: { sample: 109, pct: 25, cum_pct: 50 }
32KB: { sample: 210, pct: 49, cum_pct: 99 }
64KB: { sample: 1, pct: 0, cum_pct: 100 }
mdt.fir-MDT0002.rename_stats=
rename_stats:
- snapshot_time: 1554957753.620148602
- same_dir
4KB: { sample: 43443, pct: 75, cum_pct: 75 }
8KB: { sample: 0, pct: 0, cum_pct: 75 }
16KB: { sample: 10513, pct: 18, cum_pct: 94 }
32KB: { sample: 1518, pct: 2, cum_pct: 96 }
64KB: { sample: 1284, pct: 2, cum_pct: 99 }
128KB: { sample: 488, pct: 0, cum_pct: 99 }
256KB: { sample: 0, pct: 0, cum_pct: 99 }
512KB: { sample: 3, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 65, pct: 1, cum_pct: 1 }
8KB: { sample: 0, pct: 0, cum_pct: 1 }
16KB: { sample: 184, pct: 3, cum_pct: 4 }
32KB: { sample: 0, pct: 0, cum_pct: 4 }
64KB: { sample: 3098, pct: 60, cum_pct: 65 }
128KB: { sample: 1782, pct: 34, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 328, pct: 6, cum_pct: 6 }
8KB: { sample: 0, pct: 0, cum_pct: 6 }
16KB: { sample: 390, pct: 7, cum_pct: 13 }
32KB: { sample: 2263, pct: 44, cum_pct: 58 }
64KB: { sample: 2148, pct: 41, cum_pct: 100 }
|
|
Stephane,
That's confusing... Can you double check you're running the patch on the MDS? If this is indeed happening, then I guess there must be another gap in the patch logic.
|
|
I think so, I just checked and I did pick the correct patch set (6).
I had to port it to 2.12.0 but not very difficult. This is the patch I'm using in my 2.12.0.pl4 on both Fir's MDS:
https://gist.github.com/thiell/9d9f9046660dc9748cd695e64d1c6f21
[root@fir-md1-s1 ~]# cat /sys/fs/lustre/version
2.12.0.pl4
[root@fir-md1-s1 ~]# lctl get_param mdt.*.enable_remote_rename
mdt.fir-MDT0000.enable_remote_rename=0
mdt.fir-MDT0001.enable_remote_rename=0
mdt.fir-MDT0002.enable_remote_rename=0
mdt.fir-MDT0003.enable_remote_rename=0
[root@fir-md1-s1 ~]# lctl get_param mdt.*.rename_stats
mdt.fir-MDT0000.rename_stats=
rename_stats:
- snapshot_time: 1555006234.455267326
- same_dir
4KB: { sample: 114045, pct: 79, cum_pct: 79 }
8KB: { sample: 0, pct: 0, cum_pct: 79 }
16KB: { sample: 3444, pct: 2, cum_pct: 82 }
32KB: { sample: 7227, pct: 5, cum_pct: 87 }
64KB: { sample: 6203, pct: 4, cum_pct: 91 }
128KB: { sample: 11437, pct: 7, cum_pct: 99 }
256KB: { sample: 860, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 453, pct: 4, cum_pct: 4 }
8KB: { sample: 0, pct: 0, cum_pct: 4 }
16KB: { sample: 362, pct: 3, cum_pct: 7 }
32KB: { sample: 226, pct: 2, cum_pct: 9 }
64KB: { sample: 1, pct: 0, cum_pct: 9 }
128KB: { sample: 0, pct: 0, cum_pct: 9 }
256KB: { sample: 0, pct: 0, cum_pct: 9 }
512KB: { sample: 0, pct: 0, cum_pct: 9 }
1MB: { sample: 10000, pct: 90, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 1025, pct: 9, cum_pct: 9 }
8KB: { sample: 0, pct: 0, cum_pct: 9 }
16KB: { sample: 148, pct: 1, cum_pct: 10 }
32KB: { sample: 292, pct: 2, cum_pct: 13 }
64KB: { sample: 403, pct: 3, cum_pct: 16 }
128KB: { sample: 1103, pct: 9, cum_pct: 26 }
256KB: { sample: 2327, pct: 21, cum_pct: 47 }
512KB: { sample: 3986, pct: 36, cum_pct: 84 }
1MB: { sample: 1758, pct: 15, cum_pct: 100 }
mdt.fir-MDT0001.rename_stats=
rename_stats:
- snapshot_time: 1555006234.455585213
- same_dir
4KB: { sample: 3376, pct: 67, cum_pct: 67 }
8KB: { sample: 0, pct: 0, cum_pct: 67 }
16KB: { sample: 173, pct: 3, cum_pct: 71 }
32KB: { sample: 158, pct: 3, cum_pct: 74 }
64KB: { sample: 1268, pct: 25, cum_pct: 100 }
- crossdir_src
4KB: { sample: 113, pct: 8, cum_pct: 8 }
8KB: { sample: 0, pct: 0, cum_pct: 8 }
16KB: { sample: 0, pct: 0, cum_pct: 8 }
32KB: { sample: 421, pct: 30, cum_pct: 38 }
64KB: { sample: 859, pct: 61, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 416, pct: 29, cum_pct: 29 }
8KB: { sample: 0, pct: 0, cum_pct: 29 }
16KB: { sample: 330, pct: 23, cum_pct: 53 }
32KB: { sample: 638, pct: 45, cum_pct: 99 }
64KB: { sample: 9, pct: 0, cum_pct: 100 }
mdt.fir-MDT0002.rename_stats=
rename_stats:
- snapshot_time: 1555006234.455769900
- same_dir
4KB: { sample: 100705, pct: 76, cum_pct: 76 }
8KB: { sample: 0, pct: 0, cum_pct: 76 }
16KB: { sample: 13872, pct: 10, cum_pct: 87 }
32KB: { sample: 13518, pct: 10, cum_pct: 97 }
64KB: { sample: 1709, pct: 1, cum_pct: 98 }
128KB: { sample: 1439, pct: 1, cum_pct: 99 }
256KB: { sample: 0, pct: 0, cum_pct: 99 }
512KB: { sample: 3, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 76, pct: 1, cum_pct: 1 }
8KB: { sample: 0, pct: 0, cum_pct: 1 }
16KB: { sample: 184, pct: 2, cum_pct: 3 }
32KB: { sample: 425, pct: 5, cum_pct: 9 }
64KB: { sample: 3098, pct: 43, cum_pct: 53 }
128KB: { sample: 3303, pct: 46, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 453, pct: 6, cum_pct: 6 }
8KB: { sample: 0, pct: 0, cum_pct: 6 }
16KB: { sample: 528, pct: 7, cum_pct: 13 }
32KB: { sample: 2436, pct: 34, cum_pct: 48 }
64KB: { sample: 2148, pct: 30, cum_pct: 78 }
128KB: { sample: 1521, pct: 21, cum_pct: 100 }
mdt.fir-MDT0003.rename_stats=
rename_stats:
- snapshot_time: 1555006234.456015071
- same_dir
4KB: { sample: 1683, pct: 88, cum_pct: 88 }
8KB: { sample: 0, pct: 0, cum_pct: 88 }
16KB: { sample: 111, pct: 5, cum_pct: 94 }
32KB: { sample: 31, pct: 1, cum_pct: 96 }
64KB: { sample: 70, pct: 3, cum_pct: 99 }
128KB: { sample: 0, pct: 0, cum_pct: 99 }
256KB: { sample: 4, pct: 0, cum_pct: 100 }
- crossdir_src
4KB: { sample: 115, pct: 28, cum_pct: 28 }
8KB: { sample: 0, pct: 0, cum_pct: 28 }
16KB: { sample: 0, pct: 0, cum_pct: 28 }
32KB: { sample: 287, pct: 71, cum_pct: 100 }
- crossdir_tgt
4KB: { sample: 115, pct: 28, cum_pct: 28 }
8KB: { sample: 0, pct: 0, cum_pct: 28 }
16KB: { sample: 0, pct: 0, cum_pct: 28 }
32KB: { sample: 287, pct: 71, cum_pct: 100 }
(all 4 MDTs are currently running on fir-md1-s1)
|
|
Hm, thanks for double checking. Your patch looks just fine. Still a mystery, then.
|
|
Could it be an issue with the stats?
It has been running ok last night, still ok at the moment, but I just noticed two call traces (one with mdt_dom_discard_data). No apparent hang however. Just in case:
|
|
An issue with the stats is a good guess, but I checked that. The only place the rename stats are modified is this function:
Which is only called from mdt_reint_rename, and only (as far as I can see) in the section we skip when we recognize this case and return -EXDEV.
Hmm, so we have an explanation for your other MDS deadlock... Since applying the most recent version of the patch, have you seen the rename deadlock specifically?
The discard data hung threads thing you're reporting is concerning... Could you dump stack for all your threads, perhaps? Maybe we can see if others are also in that state. Though it looks like it may have evicted and unwound already.
|
|
So far, no Patrick, I cannot confirm that we've seen the same deadlock yet (like we did 3 days ago before the patch).
Yep I've seen your updates for yesterday's MDS deadlock and wanted to thank you for looking at it so quickly. This sounds like a very good lead.
And OK - I'm going to dump the stack for all threads when I noticed new call traces, before possible evictions. This happens pretty often.
|
|
Note that the stats do not have any specific details for remote renames, only same-dir and cross-dir renames. With DNE1 the same-dir renames would not be remote, but with DNE2 it would be possible that even same-dir renames could be remote. It probably makes sense to update the stats to include local vs. remote rename stats.
|
|
Oh I finally got it!! Thanks for explaining twice Andreas So for us, cross-dir renames should be common and normal, because all subdirectories within a same /fir/users/$u or /fir/groups/$g are on the same MDT, and it's perfectly normal that users move stuff across subdirs. All same-dirs should be on the same MDT too, we only use DNE2 in the base directories /fir/users and /fir/groups and we don't do any rename there. After that, we might have occasional rename between MDTs (remote renames), but now we have this -EXDEV patch to avoid that.
And yes, it would be great to have a way to track remote renames indeed!
At this point, to help validating the -EXDEV patch, what I will do is report back if I see any call trace with mdt_remote_object_lock. But right now I don't see any.
|
|
Ack, reading comprehension fail on my part! Thank you for pointing that out, Andreas. Duh.
So actually, no reason to doubt the updated patch. That's good.
|
|
Hi Patrick,
Our 2.12 filesystem just became partially unresponsive and I can see the following traces on the MDS:
So I dumped all tasks using sysrq-t
Please find logs + sysrq traces in fir-md1-s1_20190412.log
I may need to restart all MDTs if it doesn't recover by itself.
|
|
If you're able, dumping the lock tables could be helpful. I'll see if I can take a quick look...
|
|
dumping the lock tables now
|
|
dk.fir-md1-s1.gz
|
|
I'm about to restart all MDTs, but it's interesting that some of the latest traces still contain mdt_remote_object_lock:
|
|
Attached a few seconds of full debug (-1) just in case as dk-full.fir-md1-s1.gz
It is back online
There are already a few call traces though:
We'll see...
|
|
I restarted again all MDTs after setting enable_remote_rename=0 permanently using
instead of a "late" tuning. It's much quieter now, but we'll see.
|
|
The OSP call trace suggests possible OST issues. Something to check for.
|
|
Those could actually be seen only shortly after I started the MDTs, so perhaps due to the load when they reconnect to the OSTs.
The definitively more pressing issue I guess is to find out why we got these very often:
|
|
Stephane, I think it is result of LU-11359 issue, we were discussing already patch https://review.whamcloud.com/34071, I am updating it to handle non-updated clients gracefully so you will be able to apply patch in any order on servers/clients. It will be ready today I think
|
|
Mike, thanks for the update! This looks like it, yes. We'll likely start to apply the patch on the servers first. If we can do that sometimes next week that would be great, so let me know when you're confident with this patch.
|
|
Stephane, the current patch is not quite ready, since it doesn't properly check whether the client has the fix. Hopefully Mike will update the patch with a proper feature fix soon.
|
|
Hi Andreas,
Thanks, I'll wait for you to confirm that the patch is ready.
But this is quite urgent, as we have more and more users reporting simple mv commands hanging.
One recent report was just a move of a directory into another one, on the same MDT.
The report was that the following command was hanging:
I checked and all directories are on the same MDT:
The mv still hasn't succeeded at that time. This is impacting research work. We end up having no other solution but to recommend using other filesystems (either our NFS appliance or Oak which is running 2.10).
I see plenty of backtraces on the MDS where MDT2 is located:
|
|
If I understand the commit message of Mike's patch correctly, that is likely also related to the mdt_dom_discard_data locking issue, as it says "leading to cascading timeouts for any other locks waiting on the same resource and parent directory."
After restarting the MDT, the mv command completed. I think we need to perform some scheduled MDT restart for now, until the patch is ready.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34615/
Subject: LU-12037 mdt: call mdt_dom_discard_data() after rename unlock
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: d557ce56bee3267fbe7b29a47e4ce7a99062e42c
|
|
More locking issues today (was pretty quiet the last two days though, but we restart all MDTs every night). Is Patch Set 11 ready and if yes, can I upgrade the servers first? Will the issue only be fixed when clients+servers are all up-to-date?
|
|
Stephane,
I would say it's ready. I hope tappro will correct me if I'm wrong, but in the interests of getting you unblocked on trying the patch, my understanding is: You can upgrade the servers first - they now recognize older clients and change behavior accordingly. The issue will not be fully resolved until everything is upgraded, yeah.
|
|
Hi Patrick,
Thanks I'll try to apply the patch for the DOM locks.
This morning however, we have seen this remote lock issue again, you told me to report any new issue related to that so here it is...
first of all, as a reminder, we're running with enable_remote_rename=0
I was able to reproduce the issue, mv easily hangs (sh-ln01 is 10.9.0.61@o2ib4):
Then cd from another node hangs too:
sh-ln01 can be found there:
However, /fir/users/sthiell is on MDT0:
Which is weird because the backtrace seems to come from MDT1 or 3. Anyway, something is still wrong. So I'm thinking to fix the DOM locks first then we can see if we still see the remote_lock problem...
|
|
Wait, you can reproduce the issue with just mv? Can you describe the directory the mv is happening in? DNE involved somehow?
Upon reflection, I'm guessing you're only able to reproduce it with mv after you've already got a problem?
The trace you're showing here isn't a problem - In this case, the secondary MDT needs the rename lock. The entire file system has one rename lock, and it's on MDT0. (mdt_rename_lock is asking for that). So this sequence is expected even with Lai's patch.
Hm, can you give getdirstripe of "users" and of the directory you're trying to mv?
|
|
Yes yes sorry, I was able to reproduce a hanging rename because the filesystem was already in a weird state... I was seeing traces on both MDS already (but not any recent mdt_dom_discard_data-related traces).
|
|
|
|
After restarting the MDTs, the mv has completed:
|
|
And then, after the recovery of the MDTs, the only backtraces I can see are related to the mdt_dom_discard_data (see below). So again, I think we need to focus on fixing this issue first. But the fact that we could see the mdt_remote_object_lock_try is disturbing to me.
|
|
Patrick, the patch https://review.whamcloud.com/#/c/34071/11/ does apply with a few minor changes, but it doesn't compile against 2.12.0 because:
in ldlm_request.c, there is a reference to ldlm_is_granted() that doesn't seem to exist. I might need some help for that part.
I might need some help for the porting of this patch to 2.12
|
|
Stephane,
You can replace ldlm_is_granted(lock) with (lock->l_req_mode == lock->l_granted_mode).
About the remote lock:
If, as in the case from earlier, it's inside mdt_rename_lock and it's on a secondary MDT (not MDT0), then it's not an issue. [Edit: Sorry, too brief....] It's just trying to get the rename lock on MDT0. That does indicate some sort of hang if you're getting back traces, but it doesn't actually indicate a remote rename. There is only a single rename lock ("big file system lock" or BFL, as Andreas labels it) for the whole file system. This is kind of standard (EXT4, at least, is the same) as it massively simplifies rename locking (which is still quite complicated).
|
|
Thanks!!
This does compile against my 2.12.0 branch: https://gist.github.com/thiell/ce2445aa1e37b365e50208ebbb0b5263
Let me know if that sounds ok to you. I will build new packages after lunch.
Thanks for the remote lock explanation, this is super interesting and I'm glad to hear that it doesn't necessary mean a remote rename!!
|
|
Will try the following on top of 2.12.0:
servers (fir):
clients (sherlock):
|
|
FYI, we had another "event" so we decided to update the servers and now we're slowly starting to redeploy clients, this will take a while.
Our first client with async_discard:
Thanks much for your efforts! Fingers crossed.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34410/
Subject: LU-12037 mdt: add option for cross-MDT rename
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c6f3d533542b9462b5b8df95183d80321d4d9c34
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34616/
Subject: LU-12037 mdt: add option for cross-MDT rename
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 9cbefa99f234586c1419243a4d2c136e4f405b29
|
|
ok, for release tracking purposes, let's mark this ticket resolved because both patches have landed for 2.13 and will be in 2.12.1. There seem to be no shortage of other tickets if we need further action still.
|
|
Ok no problem. Should I update in LU-11359 regarding: the DoM locking issue (mdt_dom_discard_data) then? We only applied the patch on ~400 clients so far, so the issue isn't resolved yet. Today was very difficult, a lot of issues again, we'll patch more aggressively the rest of the ~1000 clients tomorrow with https://review.whamcloud.com/#/c/34071/11/
Also, we hit LU-12120 while restarting the MDTs multiple times today while trying to resolve the DOM locking issues:
After many restart attempts, the filesystem seems to have stabilized at this point, for now...
|
|
Yes I think that switching to LU-11359 makes sense
|
|
Because mdt.*.enable_remote_rename=0 was impacting performance, as it took a very long time for some users to mv within the filesystem, we decided to set enable_remote_rename=1 again. We did that last week and we have survived until now. It was a wrong track I guess. This is the other fixes that really helped like async_discard or LU-11285 "ldlm: reprocess whole waiting queue for IBITS". I think this one can be closed as soon as LU-11285 has landed...
|
Generated at Sat Feb 10 08:56:50 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
dk-full.fir-md1-s1.gz
fir-md1-s1-kern-syslog-20190228.log