this semaphore is needed to implement "execute-once" semantics on MDS. MDS remembers the last result/XID for every client in last_rcvd file, so if the client hasn't got a reply, it can just resend the request and MDS will reconstruct the reply being aware the request has been already executed.
if we don't have this semaphore, then concurrent request will rewrite that information in last_rcvd and MDS won't be able to detect execution status, which is pretty wrong in the number of cases (as many metadata requests are not idempotent).
this is a known issue we're going to address with multislot last_rcvd at some point. it'll likely request small changes to the protocol so that the client can say which specific slot to use like NFSv4 does (if I remember correctly).
Ned, in LU-2613 there is work being done to avoid allocating a new transno to the open request from the client. It might also be possible to send multiple open requests from the client in parallel if they do not have the O_CREAT flag set. However, Alex's point is also valid - the open requests still change the state of the MDS in memory. If the client resends the open request and the MDS has already opened it and does not detect this correctly, then it will leak an open refcount and the file will not be closed or deleted when it is unlinked.
It might be possible to detect this based on the handle used for the open, but that would further increase the complexity of the recovery process, and only fix the non-O_CREAT open() case. The generic solution for allowing multiple modifying RPCs in flight is to allow multiple slots on the last_rcvd file for a single client.
The exact mechanism for this has not been designed at this point, and is not on any development roadmap. To be honest, given the glacial speed of the OpenSFS RFP process, it is unclear how any feature development projects will be started in the foreseeable future.
Alex, Andreas, thank you for filling in the details here. It sounds like there's nothing to do in the short term for this issue. Though if you are amenable, I'll submit a patch adding a comment block above the definition of struct mdc_rpc_lock to capture some of the useful information here.
Since it's just a comment, I'll post it for review here first, to avoid wasting time for hudson/maloo.
--- a/lustre/include/lustre_mdc.h +++ b/lustre/include/lustre_mdc.h @@ -69,9 +69,27 @@ struct obd_export; struct ptlrpc_request; struct obd_device; +/** + * Serializes in-flight MDT-modifying RPC requests to preserve idempotency. + * + * This mutex is used to implement execute-once semantics on the MDT. + * The MDT stores the last transaction ID and result for every client in + * its last_rcvd file. If the client doesn't get a reply, it can safely + * resend the request and the MDT will reconstruct the reply being aware + * that the request has already been executed. Without this lock, + * execution status of concurrent in-flight requests would be + * overwritten. + * + * This design limits the extent to which we can keep a full pipeline of + * in-flight requests from a single client. This limitation could be + * overcome by allowing multiple slots per client in the last_rcvd file. + */ struct mdc_rpc_lock { + /** Lock protecting in-flight RPC concurrency. */ struct mutex rpcl_mutex; + /** Intent associated with currently executing request. */ struct lookup_intent *rpcl_it; + /** Used for MDS/RPC load testing purposes. */ int rpcl_fakes; };
To be honest, given the glacial speed of the OpenSFS RFP process, it is unclear how any feature development projects will be started in the foreseeable future.
Should I open a new ticket for this? 
Patch to add above comment:
Patch landed to Master. Please let me know if more work is needed in this ticket and I will reopen.