[LU-10274] LNet event add-on function

Related — Thu, 15 Oct 2020 13:36:42 +0000

lnet eq_callback is always called with hold of spinlock, this is useful when upper layer has requirement on strict order of events for the same MD. For example, if all events/callbacks may change some status, but only the last callback (for unlinked event) can wakeup upper layer thread, this case requires the last callback strictly happen after all other events/callbacks, otherwise upper layer thread may see wrong status.

However, there are also some cases upper layer has no requirement on order of events/callbacks, then it is unnecessary for lnet to guarantee order of events.

We can add new lnet event interface to upper layer (LNetEventAddon), by calling it, user can register an add-on for current event from lnet_eq_handler_t, lnet will call this add-on function without any lock after current lnet_eq_handler_t.

In current ptlrpc, request_in_callback is the major case that can benefit from this, current request_in_callback needs to call atomic allocator to allocate ptlrpc_request, which is not good because it can easily fail when system is under memory pressure, it also generate more lock contentions between LND threads and ptlrpc threads, which is bad for performance. With this feature, most code in request_in_callback can be moved to a new function request_in_add_on, which is registered by a new lightweight version of request_in_callback, request_in_add_on will be called without any lock.

Some other event callbacks in ptlrpc can also use this feature.

Whamcloud Community JIRA

[LU-10274] LNet event add-on function