[LU-13336] Opendir with read intent

• Type: New Feature
• Resolution: Unresolved
• Priority: Minor
• Assignee: WC Triage
• Reporter: Nathan Rutman

Description

Lustre tree walking performance is fairly poor (~30K files/s) while stat() performance is decent (~200K files/s). The culprit seems to be readdir, which is a series of small file reads from the MDS. (Note readdir isn't tested by mdtest.)

While there are a number of ideas to improve tree walking performance, we think the best potential is to implement opendir with read intent. Opendir should read and return the first page of the directory along with the directory read lock. This can avoid a bunch of RPCs: open, read req, read RDMA.

For small directories (the majority), a single page may well be sufficient to include all dirents, meaning no more RPCs necessary. For larger dirs, the client can begin processing the first page while other pages are delivered from the MDS.

Ref: Cray LUS-7695