[LU-14560] Allow number of threads to grow when all existing threads are stuck

• Type: Improvement
• Resolution: Duplicate
• Priority: Major
• Affects Version/s: Lustre 2.15.0
• Assignee: WC Triage
• Reporter: Oleg Drokin

Description

Currently when we have a chokepoint of some sort (be it slow disk, a blocked ldlm resource and the like), relatively low number of threads we start by default gets consumed pretty quickly resulting in no requests being processed even those that would not block because they don't touch the contended resource.

While normally it's a good idea to maintain the thread pool small as it improves cpu cache effectiveness and also puts less of a load on a spinning disk setup (if present) when the entire thread pool is plugged for some time I think it is beneficial to actually spawn a lot more threads as a "one shot" only and we should not be any worse off wrt the end outcome. Memory consumption of server threads is relatively minor so even if you have a massive number of clients send you dozens of RPCs that results in tens of thousands of threads, modern systems should take it relatively easily.

Worst comes to worst all the new threads are also stuck, but some of them might actually progress which would be a net benefit.

Half measures are also possible to only spawn threads that would e.g. process "high priority" requests.

Issue Links

• duplicates: LU-14564 Allow number of threads to grow when all existing threads are stuck