[LU-7915] investigate heuristics for SPARK client getting MDS openlock

• Status: Open
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: Emoly Liu
• Resolution: Unresolved

Description

During discussion with LBNL on their Hadoop Spark project, they said that there was considerable overhead when running on Lustre because of repeated open+close of the same files causing extra RPC traffic to the MDS.

Lustre has the ability to cache opens on the client with a DLM openlock, but this isn't done for regular opens by default because it has extra overhead compared to uncached opens, but only for NFS opens because the knfsd repeatedly opens the same file.

It would be worthwhile to firstly implement a tunable to enable opencache on a per-client basis (LU-5426) and then measure the performance impact of this tunable for normal usage and for Spark.

Issue Links

• is related to: LU-10948 client cache open lock after N opens
• is related to: LU-10948 client cache open lock after N opens
• is related to: LU-5426 Add more controls for open file cache

Comments

[Gabriele Paciucci (Inactive)]

Hi adilger
Could you please ask more details on their Spark project? I run benchmark test on Spark and I didn't notice this issue. I notice an extensive use of getxattr() solved by the new (2.5) lustre local xattr cache. Thank you

[Joseph Gmitter (Inactive)]

Hi Emoly,

Could you please have a look at measuring the performance as indicated by Andreas' final comment:

It would be worthwhile to firstly implement a tunable to enable opencache on a per-client basis (LU-5426) and then measure the performance impact of this tunable for normal usage and for Spark.

Thanks.
Joe

[Emoly Liu]

Can anybody show me any more details about this Spark performance testing? e.g. Spark version, benchmark tool, lustre version and how large scale. Thanks.

[Andreas Dilger]

The following pages describe their testing:

http://crd.lbl.gov/departments/computer-science/CLaSS/staff/costin-iancu/intel-parallel-computing-center-big-data-support-on-hpc-systems/
http://crd.lbl.gov/assets/pubs_presos/spark-on-cray.pdf

[Gerrit Updater]

Emoly Liu ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19664
Subject: LU-7915 mdc: add a tunable to enable opencache
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 770f30f801417bcc24b39f0f58fe0abe35bc2bcd