https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15831 Lustre <p>When trying to run GPUDirect, it was found on install required software step:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ sudo ./mlnxofedinstall ... Installation passed successfully To load the <span class="code-keyword">new</span> driver, run: /etc/init.d/openibd restart $ sudo /etc/init.d/openibd restart Unloading ib_uverbs [FAILED] rmmod: ERROR: Module ib_uverbs is in use by: nv_peer_mem $ sudo rmmod nv_peer_mem $ sudo /etc/init.d/openibd restart Unloading HCA driver:[ OK ] Loading Mellanox MLX5_IB HCA driver:  [FAILED] Loading Mellanox MLX5 HCA driver: [FAILED] Loading HCA driver and Access Layer:  [FAILED] Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information and open an issue in the http:<span class="code-comment">//support.mellanox.com/SupportWeb/service_center/SelfService </span> $ sudo modprobe nv_peer_mem modprobe: FATAL: Module nv_peer_mem not found in directory /lib/modules/5.4.0-109-<span class="code-keyword">generic</span> $ sudo modprobe lustre modprobe: ERROR: could not insert <span class="code-quote">'lustre'</span>: Invalid argument</pre> </div></div> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ sudo ./mlnxofedinstall Checking SW Requirements...Removing old packages... Installing <span class="code-keyword">new</span> packages Installing ofed-scripts-5.5... Installing mlnx-tools-5.2.0... Installing mlnx-ofed-kernel-utils-5.5... Installing mlnx-ofed-kernel-dkms-5.5...Error: mlnx-ofed-kernel-dkms installation failed! Problem: mlx5_ib: module file: /lib/modules/5.4.0-105-<span class="code-keyword">generic</span>/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko, from <span class="code-keyword">package</span>: linux-modules-extra-5.4.0-105-<span class="code-keyword">generic</span>. Collecting debug info... See:     /tmp/MLNX_OFED_LINUX.1302312.logs/mlnx-ofed-kernel-dkms.debinstall.log Removing newly installed packages...</pre> </div></div> <p>This prevents gds tests from running completely:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> =========================  Platform verification error : nvidia-fs driver is not loadedSUCCESS FILESYSTEM VERSION CHECK: ofed_info: current version: MLNX_OFED_LINUX-5.5-1.0.3.2: (Supported) min version supported: MLNX_OFED_LINUX-4.6-1.0.1.1 SUCCESS nvidia-fs driver is not loadedSUCCESS usage: gdscheck.py [-h] [-p] [-f FILE] [-v] [-V]GPUDirectStorage platform checkeroptional arguments:   -h, --help  show <span class="code-keyword">this</span> help message and exit   -p          gds platform check   -f FILE     gds file check   -v          gds version checks   -V          gds fs checks SUCCESS gdscheck.py python2 tests =========================  Platform verification error : nvidia-fs driver is not loadedSUCCESS FILESYSTEM VERSION CHECK: ofed_info: current version: MLNX_OFED_LINUX-5.5-1.0.3.2: (Supported) min version supported: MLNX_OFED_LINUX-4.6-1.0.1.1 SUCCESS nvidia-fs driver is not loadedSUCCESS usage: gdscheck.py [-h] [-p] [-f FILE] [-v] [-V]GPUDirectStorage platform checkeroptional arguments:   -h, --help  show <span class="code-keyword">this</span> help message and exit   -p          gds platform check   -f FILE     gds file check   -v          gds version checks   -V          gds fs checks SUCCESS gdscheck.py current running python tests =========================  Platform verification error : nvidia-fs driver is not loadedSUCCESS FILESYSTEM VERSION CHECK: ofed_info: current version: MLNX_OFED_LINUX-5.5-1.0.3.2: (Supported) min version supported: MLNX_OFED_LINUX-4.6-1.0.1.1 SUCCESS nvidia-fs driver is not loadedSUCCESS usage: gdscheck.py [-h] [-p] [-f FILE] [-v] [-V]GPUDirectStorage platform checkeroptional arguments:   -h, --help  show <span class="code-keyword">this</span> help message and exit   -p          gds platform check   -f FILE     gds file check   -v          gds version checks   -V          gds fs checks SUCCESS ************************************************** gdscheck.py test results : 12 /  12 tests passed ************************************************** Starting basic gdsio Tests /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 4k </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 3k </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 3k -o 1 </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 2k </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 2k -o 1 </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 1k </span>SUCCESS /usr/local/gds/tools/gdsio -f  /data/sanity/tests<span class="code-comment">//sparse1G -d 0 -f /data/sanity/tests//sparse1G -d 0 -s 128K -i 1k -o 1 </span>SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 1 -o 1 </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 3 -k 1234 -o 1 </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -V -D /data/sanity/tests<span class="code-comment">// -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 1 -o 1 </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -V -D /data/sanity/tests<span class="code-comment">// -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 3 -k 1234 -o 1 </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -D /data/sanity/tests<span class="code-comment">// -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 3 -k 1234 -o 1 -F -R </span>SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 1 -o 1 -b </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 8M:32M -x 0 -I 1 -o 1 -b </span>Verifying data  SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 0 -o 1 -b </span>SUCCESS /usr/local/gds/tools/gdsio -V -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 8M:32M -x 0 -I 0 -o 1 -b </span>SUCCESS /usr/local/gds/tools/gdsio -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 32K:1024K:1K -x 0 -I 0 -o 0 -b </span>SUCCESS /usr/local/gds/tools/gdsio -f /data/sanity/tests<span class="code-comment">//sparse1G -d 0  -w 8 -s 1G -i 8M:32M -x 0 -I 0 -o 0 -b </span>SUCCESS ************************************************** gdiso tests : 18 /  18 tests passed ************************************************** Starting Offset Tests TestCase:Read odd offset /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 616  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 616  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read odd gpu offsets 1, 2, 3, 4, 4K-1, 4K, 4K+1, 60K, 64K, 68K /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 4096 -o 0  -d 0 -t 1 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 4096 -o 0  -d 0 -t 1 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 1 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 2 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 3 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 4 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 4095 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 4096 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 4097 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 4097 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 61440 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 65536 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 69632 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 1 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 2 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 3 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 4 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 4095 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 4096 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 4097 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 61440 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 65536 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 69632 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write odd size - sync /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485761 -o 4096  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write odd size - async /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485761 -o 4096  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:odd offset and odd size - sync /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485748 -o 119  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:odd offset and odd size - async /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485748 -o 119  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write 1 <span class="code-object">byte</span> from offset 0 (sync and async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 1 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 1 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write 1 <span class="code-object">byte</span> from offset 3 (sync and async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 1 -o 3  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 1 -o 3  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write big file 10G (odd size) - sync /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 220201060 -o 4096  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read/write big file 10G (odd size) - async /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 220201060 -o 4096  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read beyond EOF (read 2G on a 1G file - async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 0 -s 209714688 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read beyond EOF (read 2G on a 1G file - sync) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 1 -s 209714688 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read beyond EOF - odd size (read 2G on a 1G file - sync and async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 0 -s 209714689 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 1 -s 209714689 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read beyond EOF odd offset (read 2G on a 1G file - sync and async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 1 -s 209714688 -o 616  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1G -n 1 -m 0 -s 209714688 -o 616  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read size beyond EOF (small file) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1099 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1099 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1099 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1099 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read just <span class="code-object">short</span> of EOF /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1000 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1000 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1000 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1000 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read offset from EOF /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 999 -o 1024  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 999 -o 1024  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1 -o 1024  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1 -o 1024  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read offset beyond EOF (sync and async) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 1 -s 1 -o 1025  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparse1K -n 1 -m 0 -s 1 -o 1025  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read with odd gpu_offset /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read with odd gpu_offset and odd file offset /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 617  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 617  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read at 128k GPU page offset /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Read beyond 64k (odd gpu offset) /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 10485760 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Overwrite an existing file within EOF /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 52428800 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 52428800 -o 1  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 52428805 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 52428805 -o 0  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:Offset beyond EOF writes /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 4096 -o 4099  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 4096 -o 4099  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:offset just <span class="code-object">short</span> of EOF writes /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 3 -o 4094  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 3 -o 4094  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED TestCase:offset from EOF writes /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 0 -s 4096 -o 3  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED /usr/local/gds/tools/gdsio_verify  -f /data/sanity/tests<span class="code-comment">//sparserandom_big -n 1 -m 1 -s 4096 -o 3  -d 0 -t 0 -p 0 </span>file register error: nvidia-fs driver is not loaded FAILED ************************************************** File offset and GPU Buffer offset Tests : 0 /  73 tests passed ************************************************** running cufile sample tests sample 1 FAILED sample 2 opening file /data/sanity/tests<span class="code-comment">//sparse1G_sample2 </span>FAILED sample 3 FAILED sample 4 FAILED sample 5 FAILED sample 6 FAILED sample 7 FAILED sample 8 PASS: cufile success status:Success SUCCESS sample 14 opening file /data/sanity/tests<span class="code-comment">//sparse1G </span>FAILED sample 15 FAILED ************************************************** cufile sample tests : 1 /  10 tests passed ************************************************** Testing gdscp functionality /usr/local/gds/tools/gdscp /data/sanity/tests<span class="code-comment">//sparse1G /data/sanity/tests//sparse1G_copy 0 -v </span>file register error: nvidia-fs driver is not loaded FAILED ************************************************** gdscp tests : 0 /  1 tests passed ************************************************** Testing Batch State Machine /usr/local/gds/tools<span class="code-comment">//tests/cufile_batch_test_state_machine /data/sanity/tests//sparse1G 0 &amp;&amp; pass || fail </span>FAILED /usr/local/gds/tools<span class="code-comment">//tests/cufile_batch_test_state_machine /data/sanity/tests//sparse1G 1 &amp;&amp; pass || fail </span>FAILED /usr/local/gds/tools<span class="code-comment">//tests/cufile_batch_test_state_machine /data/sanity/tests//sparse1G 2 &amp;&amp; pass || fail </span>FAILED /usr/local/gds/tools<span class="code-comment">//tests/cufile_batch_test_state_machine /data/sanity/tests//sparse1G 3 &amp;&amp; pass || fail </span>FAILED /usr/local/gds/tools<span class="code-comment">//tests/cufile_batch_test_state_machine /data/sanity/tests//sparse1G 4 &amp;&amp; pass || fail </span>FAILED ************************************************** Batch State Machine Tests : 0 /  5 tests passed ************************************************** Performing cufile API tests /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 4 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 5 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 6 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 7 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 8 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 9 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufregister 10 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 4 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testbufderegister 5 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testclosefd /data/sanity/tests//sparse1G 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testclosefd /data/sanity/tests//sparse1G 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testclosefd /data/sanity/tests//sparse1G 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testclosefd /data/sanity/tests//sparse1G 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriver 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriver 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriver 2 </span>cufile driver close: nvidia-fs driver is not loaded SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 1 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 2 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 3 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 4 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenfd  /data/sanity/tests//sparse1G 5 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_rw  /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G_VERIFY 0  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_rwmanaged  /data/sanity/tests//sparse1G 0  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_rw_unreg  /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G_VERIFY 0 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_rw_unreg  /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G_VERIFY 0 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_rw_unreg  /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G_VERIFY 0 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 4 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 5 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 6 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 7 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 8 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testopenflags /data/sanity/tests//sparse1G 9 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -p 8 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -b 8 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -d 8 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -c 8 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -b 1024 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testdriverprops -d 1024 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_driver_close /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_driver_close /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G 1 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_driver_close /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G 2 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_io_race /data/sanity/tests//sparse1G  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testvalidnvbuf  /data/sanity/tests//sparse1G 0  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testvalidnvbuf  /data/sanity/tests//sparse1G 0  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_driver_close /data/sanity/tests//sparse1G /data/sanity/tests//sparse1G 3 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_io_race /data/sanity/tests//sparse1G  </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_invalid_write /data/sanity/tests//sparse1G 0 0 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_invalid_write /data/sanity/tests//sparse1G 0 1 </span>SUCCESS /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_invalid_offsets /data/sanity/tests//sparse1G 0 0 </span>FAILED /usr/local/gds/tools<span class="code-comment">//api_tests/cufile_testcudacontext_switch /data/sanity/tests//sparse_CTX_VERIFY 0 </span>FAILED End: nvidia-fs: GDS Version: 1.2.1.4  NVFS statistics(ver: 4.0) NVFS Driver(version: 2.11.0) Mellanox PeerDirect Supported: True IO stats: Disabled, peer IO stats: Disabled Logging level: infoActive Shadow-Buffer (MiB): 0 Active <span class="code-object">Process</span>: 0 Reads                : err=0 io_state_err=0 Sparse Reads                : n=0 io=0 holes=0 pages=0  Writes                : err=0 io_state_err=0 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0 Mmap                : n=0 ok=0 err=0 munmap=0 Bar1-map            : n=0 ok=0 err=0 free=0 callbacks=0 active=0 Error                : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0 Ops                : Read=0 Write=0 BatchIO=0 ************************************************** API Tests, : 10 /  63 tests passed ************************************************** Testsuite : 41 / 182 tests passed done tests:Mon May 2 16:48:42 UTC 2022 </pre> </div></div> <p>It seems that a patch <a href="https://review.whamcloud.com/#/c/45327/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/45327/</a> needs to be applied to the master.</p> <p>With MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64 and MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64 the result is the same.</p> <p> </p> <p> </p> LU-15831

Lustre 2.15 client breaks DGXA100 MOFED

Bug Major Resolved Fixed Gaurang Tapase Oleg Kulachenko debian ubuntu Fri, 6 May 2022 10:23:52 +0000 Wed, 25 May 2022 06:43:57 +0000 Wed, 25 May 2022 06:18:09 +0000 0 5 <p>Minh</p> <p>Please can you advise</p> <p>Thanks</p> <p>Peter</p> <p>this is caused by <a href="https://jira.whamcloud.com/browse/LU-12019" title="Recognize Debian Kernel in autoconf and unify kernel module package directory" class="issue-link" data-issue-key="LU-12019"><del>LU-12019</del></a> <a href="https://review.whamcloud.com/34329" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34329</a></p> <p>Ok. Can we revert that change for 2.15.0 and then take longer to assess how to support that change without introducing a regression for other usage?</p> <p>revert patch <a href="https://review.whamcloud.com/47238" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47238</a></p> <p>Landed for 2.15</p> <p>After reinstall Lustre mlnxofedinstall works without errors.</p> <p>Logs: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/43593/43593_mlnx_logs.txt" title="mlnx_logs.txt attached to LU-15831">mlnx_logs.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>But:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ddn@a100-01:~/okulachenko/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ sudo lustre_rmmod [sudo] password <span class="code-keyword">for</span> ddn: ddn@a100-01:~/okulachenko/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ sudo /etc/init.d/openibd restart Unloading HCA driver:                                      [  OK  ] Loading Mellanox MLX5_IB HCA driver:                       [FAILED] Loading Mellanox MLX5 HCA driver:                          [FAILED] Loading HCA driver and Access Layer:                       [FAILED] Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information and open an issue in the http:<span class="code-comment">//support.mellanox.com/SupportWeb/service_center/SelfService</span></pre> </div></div> <p>I think that we found a new issue. We will re-image A100 and run firmware upgrade. Most likely the new issue is not in the Lustre, but in the firmware.</p> <p> </p> <p> </p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mdiep" class="user-hover" rel="mdiep">mdiep</a> It seems that the fix did not help.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ lctl get_param version version=2.15.0_RC3_2_g7905359 ddn@a100-01:~/okulachenko/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64$ sudo ./mlnxofedinstall --add-kernel-support --distro ubuntu20.04 Note: This program will create MLNX_OFED_LINUX TGZ <span class="code-keyword">for</span> ubuntu20.04 under /tmp/MLNX_OFED_LINUX-5.5-1.0.3.2-5.4.0-109-<span class="code-keyword">generic</span> directory. See log file /tmp/MLNX_OFED_LINUX-5.5-1.0.3.2-5.4.0-109-<span class="code-keyword">generic</span>/mlnx_iso.3333249_logs/mlnx_ofed_iso.3333249.logChecking <span class="code-keyword">if</span> all needed packages are installed... Building MLNX_OFED_LINUX DEBS . Please wait... Creating metadata-rpms <span class="code-keyword">for</span> 5.4.0-109-<span class="code-keyword">generic</span> ... WARNING: If you are going to configure <span class="code-keyword">this</span> <span class="code-keyword">package</span> as a repository, then please note WARNING: that it is not signed, therefore, you need to set <span class="code-quote">'trusted=yes'</span> in the sources.list file. WARNING: Example: deb [trusted=yes] file:/&lt;path to MLNX_OFED DEBS folder&gt; ./ Created /tmp/MLNX_OFED_LINUX-5.5-1.0.3.2-5.4.0-109-<span class="code-keyword">generic</span>/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-ext.tgz Removing old packages... Uninstalling the previous version of MLNX_OFED_LINUX Installing /tmp/MLNX_OFED_LINUX-5.5-1.0.3.2-5.4.0-109-<span class="code-keyword">generic</span>/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-ext /tmp/MLNX_OFED_LINUX-5.5-1.0.3.2-5.4.0-109-<span class="code-keyword">generic</span>/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-ext/mlnxofedinstall --force --without-dkms --distro ubuntu20.04 Logs dir: /tmp/MLNX_OFED_LINUX.3838794.logs General log file: /tmp/MLNX_OFED_LINUX.3838794.logs/general.logBelow is the list of MLNX_OFED_LINUX packages that you have chosen (some may have been added by the installer due to <span class="code-keyword">package</span> dependencies):ofed-scripts mlnx-tools mlnx-ofed-kernel-utils mlnx-ofed-kernel-modules iser-modules isert-modules srp-modules rdma-core libibverbs1 ibverbs-utils ibverbs-providers libibverbs-dev libibverbs1-dbg libibumad3 libibumad-dev ibacm librdmacm1 rdmacm-utils librdmacm-dev mstflint ibdump libibmad5 libibmad-dev libopensm opensm opensm-doc libopensm-devel libibnetdisc5 infiniband-diags mft kernel-mft-modules perftest ibutils2 ar-mgr dump-pr ibsim ibsim-doc ucx sharp hcoll openmpi mpitests knem-modules libdapl2 dapl2-utils libdapl-dev dpcp srptools mlnx-ethtool mlnx-iproute2 rshimThis program will install the MLNX_OFED_LINUX <span class="code-keyword">package</span> on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, <span class="code-keyword">do</span> not reinstall them.Checking SW Requirements... Removing old packages... Installing <span class="code-keyword">new</span> packages Installing ofed-scripts-5.5... Installing mlnx-tools-5.2.0... Installing mlnx-ofed-kernel-utils-5.5... Installing mlnx-ofed-kernel-modules-5.5... Installing iser-modules-5.5... Installing isert-modules-5.5... Installing srp-modules-5.5... Installing rdma-core-55mlnx37... Installing libibverbs1-55mlnx37... Installing ibverbs-utils-55mlnx37... Installing ibverbs-providers-55mlnx37... Installing libibverbs-dev-55mlnx37... Installing libibverbs1-dbg-55mlnx37... Installing libibumad3-55mlnx37... Installing libibumad-dev-55mlnx37... Installing ibacm-55mlnx37... Installing librdmacm1-55mlnx37... Installing rdmacm-utils-55mlnx37... Installing librdmacm-dev-55mlnx37... Installing mstflint-4.16.0... Installing ibdump-6.0.0... Installing libibmad5-55mlnx37... Installing libibmad-dev-55mlnx37... Installing libopensm-5.10.0.MLNX20211115.e645cc83... Installing opensm-5.10.0.MLNX20211115.e645cc83... Installing opensm-doc-5.10.0.MLNX20211115.e645cc83... Installing libopensm-devel-5.10.0.MLNX20211115.e645cc83... Installing libibnetdisc5-55mlnx37... Installing infiniband-diags-55mlnx37... Installing mft-4.18.0... Installing kernel-mft-modules-4.18.0... Installing perftest-4.5... Installing ibutils2-2.1.1... Installing ar-mgr-1.0... Installing dump-pr-1.0... Installing ibsim-0.10... Installing ibsim-doc-0.10... Installing ucx-1.12.0... Installing sharp-2.6.1.MLNX20211124.aac4a56... Installing hcoll-4.7.3202... Installing openmpi-4.1.2rc2... Installing mpitests-3.2.20... Installing knem-modules-1.1.4.90mlnx1... Installing libdapl2-2.1.10.1.mlnx... Installing dapl2-utils-2.1.10.1.mlnx... Installing libdapl-dev-2.1.10.1.mlnx... Installing dpcp-1.1.17... Installing srptools-55mlnx37... Installing mlnx-ethtool-5.13... Installing mlnx-iproute2-5.14.0... Installing rshim-2.0.6... Selecting previously unselected <span class="code-keyword">package</span> mlnx-fw-updater. (Reading database ... 224924 files and directories currently installed.) Preparing to unpack .../mlnx-fw-updater_5.5-1.0.3.2_amd64.deb ... Unpacking mlnx-fw-updater (5.5-1.0.3.2) ... Setting up mlnx-fw-updater (5.5-1.0.3.2) ...Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.confInitializing... Attempting to perform Firmware update... Querying Mellanox devices firmware ... Querying Mellanox devices firmware ... Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  0c:00.0   Base GUID:        0c42a10300555aaa   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/lJcLV6m0FI Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  12:00.0   Base GUID:        0c42a10300555dee   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/paBqhmZPH7 Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  4b:00.0   Base GUID:        043f720300f55646   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/ovxoNfgM6c Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  54:00.0   Base GUID:        0c42a10300555dbe   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/FZHvz2eu6S Querying Mellanox devices firmware ... Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653106A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; dual-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000225   PCI Device Name:  61:00.0   Base MAC:         1c34da6c9046   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/J6qmeEZp3kDevice #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  8d:00.0   Base GUID:        0c42a10300555d62   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/i3Ih6BBFkg Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  94:00.0   Base GUID:        0c42a10300555afe   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/ITvKfCj1dJ Querying Mellanox devices firmware ...Device #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  ba:00.0   Base GUID:        0c42a10300555af6   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/Q8rB0zMS9EDevice #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653105A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; single-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000223   PCI Device Name:  cc:00.0   Base GUID:        0c42a10300555e02   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/7EZbD0vG0HDevice #1: ----------  Device Type:      ConnectX6   Part <span class="code-object">Number</span>:      MCX653106A-HDA_Ax   Description:      ConnectX-6 VPI adapter card; HDR IB (200Gb/s) and 200GbE; dual-port QSFP56; PCIe4.0 x16; tall bracket; ROHS R6   PSID:             MT_0000000225   PCI Device Name:  e1:00.0   Base MAC:         0c42a11b7dee   Versions:         Current        Available      FW             20.33.1048     20.32.1010      PXE            3.6.0502       3.6.0502      UEFI           14.26.0017     14.25.0017  Status:           Up to date Log File: /tmp/gQkEPSV1Mp Real log file: /tmp/MLNX_OFED_LINUX.3838794.logs/fw_update.log Device (0c:00.0):     0c:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (12:00.0):     12:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (4b:00.0):     4b:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (54:00.0):     54:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (61:00.0):     61:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (61:00.1):     61:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (8d:00.0):     8d:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (94:00.0):     94:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (ba:00.0):     ba:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (cc:00.0):     cc:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (e1:00.0):     e1:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sDevice (e1:00.1):     e1:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]     Link Width: x16     PCI Link Speed: 16GT/sInstallation passed successfully To load the <span class="code-keyword">new</span> driver, run: /etc/init.d/openibd restart ddn@a100-01:~/okulachenko/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64$ sudo /etc/init.d/openibd restart Unloading HCA driver:                                      [  OK  ] Loading Mellanox MLX5_IB HCA driver:                       [FAILED] Loading Mellanox MLX5 HCA driver:                          [FAILED] Loading HCA driver and Access Layer:                       [FAILED]Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information and open an issue in the http:<span class="code-comment">//support.mellanox.com/SupportWeb/service_center/SelfService </span> ddn@a100-01:~/okulachenko/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64$ sudo modprobe -v lustre insmod /lib/modules/5.4.0-109-<span class="code-keyword">generic</span>/updates/kernel/net/libcfs.ko cpu_npartitions=32 insmod /lib/modules/5.4.0-109-<span class="code-keyword">generic</span>/updates/kernel/net/lnet.ko networks=<span class="code-quote">"o2ib(ibp12s0,ibp18s0,enp225s0f1,ibp75s0,ibp84s0,enp97s0f1,ibp141s0,ibp148s0,ibp186s0,ibp204s0)"</span> lnet_transaction_timeout=100 lnet_retry_count=2 insmod /lib/modules/5.4.0-109-<span class="code-keyword">generic</span>/updates/kernel/fs/obdclass.ko insmod /lib/modules/5.4.0-109-<span class="code-keyword">generic</span>/updates/kernel/fs/ptlrpc.ko modprobe: ERROR: could not insert <span class="code-quote">'lustre'</span>: Network is down</pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=gtapase" class="user-hover" rel="gtapase">gtapase</a> please take a look</p> <p>Gaurang, any update on this issue? This is one of the few remaining issues before the 2.15.0 release.</p> <p>I don't have access to DGX A100 system, but I tried compiling lustre 2.15 on a ubuntu 20.04 system with MoFED 5.5 installed and it worked fine. I could load the lustre module as well.</p> <p>Is this something specific to DGX A100?</p> <p>Just built the master branch of fs/lustre-release on DGX A100 and could load the lustre module as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@a100-01:/home/ddn/gtapase/exa-client# dpkg -l | grep lustre rc  lustre-client-modules-5.14.0-1032-oem             2.14.0-ddn39-11-g767352e-1              amd64        Lustre Linux kernel module (kernel 5.14.0-1032-oem) ii  lustre-client-modules-5.4.0-109-generic           2.15.50-13-gc524079-dirty-1             amd64        Lustre Linux kernel module (kernel 5.4.0-109-generic) rc  lustre-client-modules-5.4.0-96-generic            2.15.0-RC3-2-g7905359-1                 amd64        Lustre Linux kernel module (kernel 5.4.0-96-generic) ii  lustre-client-utils                               2.15.50-13-gc524079-dirty-1             amd64        Userspace utilities for the Lustre filesystem (client) ii  lustre-dev                                        2.15.50-13-gc524079-dirty-1             amd64        Development files for the Lustre filesystem rc  lustre-source                                     2.15.0-RC3-2-g7905359-1                 all          source for Lustre filesystem client kernel modules </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@a100-01:/home/ddn/gtapase/exa-client# lsmod | grep lustre lustre               1007616  0 lmv                   212992  1 lustre mdc                   274432  1 lustre lov                   331776  2 mdc,lustre ptlrpc               1355776  7 fld,osc,fid,lov,mdc,lmv,lustre obdclass             3297280  8 fld,osc,fid,ptlrpc,lov,mdc,lmv,lustre lnet                  659456  6 osc,ko2iblnd,obdclass,ptlrpc,lmv,lustre libcfs                245760  11 fld,lnet,osc,fid,ko2iblnd,obdclass,ptlrpc,lov,mdc,lmv,lustre </pre> </div></div> <p> </p> <p>I'm trying to run tests.<br/> Now this error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /usr/local/cuda-11.5/gds/tools/gdscheck.py -p  cuInit Failed, error CUDA_ERROR_SYSTEM_NOT_READY  cuFile initialization failed  Platform verification error : CUDA Driver API error </pre> </div></div> <p>But these are gds tools problems, not Lustre</p> <p>Updating cuda to the latest version fixed it.</p> Related LU-12019 Development Rank 1|i02p6f: Rank (Obsolete) 9223372036854775807 Severity