[LU-19848] Implement p2pdma support for Lustre

• Type: Improvement
• Resolution: Unresolved
• Priority: Medium
• Severity: 3
• Assignee: Tim Day
• Reporter: Tim Day

Description

Currently, Lustre supports GPUDirect Storage for NVIDIA devices via the out-of-tree GDS driver [1], that interfaces with the out-of-tree NVIDIA display driver. Recent versions of cuFile and CUDA support p2pdma [1] as an alternative to the out-of-tree GDS driver [2].

 

Lustre should adopt p2pdma as the primary method to support GPUDirect Storage. This enables us to drop the dependency on the out-of-tree GDS driver - which is a requirement for upstreaming Lustre into the Linux kernel. Furthermore, it allows us to increase device support beyond only NVIDIA [4].

 

[1] https://docs.kernel.org/driver-api/pci/p2pdma.html

[2] https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html#troubleshooting-and-faq-for-nvme-support-using-linux-pci-p2pdma

[3]https://github.com/NVIDIA/gds-nvidia-fs

[4] https://github.com/ROCm/hipFile/README.md

 

 

Issue Links

• is related to: LU-12511 Prepare lustre for adoption into the linux kernel