[LU-9114] Make MDS (And other server threads?) hog CPU less Created: 14/Feb/17 Updated: 17/Dec/20 Resolved: 17/Dec/20 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Lustre 2.14.0 |
| Type: | Improvement | Priority: | Minor |
| Reporter: | Oleg Drokin | Assignee: | Andreas Dilger |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
It's somewhat widely seen in various logs that pacemaker complaints its thread was not scheduled for tens of seconds which is way too excessive. There are also some bandaids discussed like using numa settings to cordon off one cpu from use by Lustre, but those are just that - bandaids. We probably can play with various debug settings that warn about this and make the timeouts lower to try and catch more of the offenders. Likely have a bunch in flock code with its double loops |
| Comments |
| Comment by Andreas Dilger [ 20/Apr/17 ] |
|
In addition to checking if any one MDS thread running too long without scheduling, it may also be that the many MDS kernel threads are scheduled with a higher priority and prevent the userspace threads from being run. I think for pacemaker and such, it makes sense to mlock() the heartbeat daemons into memory (so they aren't swapped) and run them with realtime priority (or something like nice -15) so that they can always get CPU time even when all of the MDS threads are running. |
| Comment by Peter Jones [ 14/Dec/17 ] |
|
Dmitry Can you please investigate this area as a longer term task for 2018 Peter |
| Comment by Gerrit Updater [ 17/Jul/20 ] |
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/39435 |
| Comment by Gerrit Updater [ 17/Dec/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39435/ |
| Comment by Peter Jones [ 17/Dec/20 ] |
|
Landed for 2.14 |