https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1307 Lustre <p>Customer reports that some clients have difficulties mounting Lustre filesystems. Running lustre_rmmod then mount -at lustre seemes to clear up the problem. This is right after a reboot of the system.</p> <p><span class="error">&#91;root@dtn1 ~&#93;</span># mount -at lustre<br/> mount.lustre: mount 10.174.80.40@o2ib2:10.174.80.41@o2ib2:/scratch1 at /scratch1 failed: No such device<br/> Are the lustre modules loaded?<br/> Check /etc/modprobe.conf and /proc/filesystems<br/> Note 'alias lustre llite' should be removed from modprobe.conf<br/> mount.lustre: mount 10.174.80.42@o2ib2:10.174.80.43@o2ib2:/scratch2 at /scratch2 failed: No such device<br/> Are the lustre modules loaded?<br/> Check /etc/modprobe.conf and /proc/filesystems<br/> Note 'alias lustre llite' should be removed from modprobe.conf<br/> <span class="error">&#91;root@dtn1 ~&#93;</span># lustre_rmmod<br/> <span class="error">&#91;root@dtn1 ~&#93;</span># mount -at lustre<br/> <span class="error">&#91;root@dtn1 ~&#93;</span># df -h<br/> Filesystem Size Used Avail Use% Mounted on<br/> /dev/mapper/vg_dtn1-lv_root<br/> 50G 17G 31G 36% /<br/> tmpfs 24G 0 24G 0% /dev/shm<br/> /dev/sda1 485M 52M 408M 12% /boot<br/> 10.174.80.40@o2ib2:10.174.80.41@o2ib2:/scratch1<br/> 2.5P 288T 2.2P 12% /scratch1<br/> 10.174.80.42@o2ib2:10.174.80.43@o2ib2:/scratch2<br/> 3.1P 427T 2.7P 14% /scratch2</p> <p>/etc/fstab:<br/> ...<br/> 10.174.80.40@o2ib2:10.174.80.41@o2ib2:/scratch1 /scratch1 lustre defaults,flock 0 0<br/> 10.174.80.42@o2ib2:10.174.80.43@o2ib2:/scratch2 /scratch2 lustre defaults,flock 0 0</p> <p><span class="error">&#91;root@dtn1 ~&#93;</span># cat /etc/modprobe.d/lustre.conf </p> <ol> <li>Lustre module configuration file<br/> options lnet networks="o2ib2(ib0)"</li> </ol> <p>Also, I have attached /var/log/messages showing the recent boot and the lustre errors reported.</p> <p>You can see in the log that I ran mount -at lustre at Apr 11 13:14:20.<br/> Then I ran lustre_rmmod and mount -at lustre and it worked.</p> <p>The customer is asking why this is happening and I do not have an explanation.<br/> I encountered similar issues on other clients after a reboot of the entire system.</p> Servers: CentOS 5.5<br/> Clients: RHEL 6.0 LU-1307

Clients having issues mounting Lustre

Bug Minor Resolved Cannot Reproduce Doug Oucharek Dennis Nelson Wed, 11 Apr 2012 09:45:38 +0000 Mon, 29 May 2017 04:08:58 +0000 Mon, 29 May 2017 04:08:58 +0000 Lustre 1.8.6 0 3 <p>Are you certain the servers have finished recovery after the reboot? Please examine the client system log, there should be LustreErrors there which may provide more information</p> <p>Yes, I am sure that recovery was complete. The servers were booted yesterday and have been back in production for over 12 hours. I included the messages file on the original post, it had lustre errors in it.</p> <p>I believe that what might be happening is that the system is attempting to mount the filesystems before the IB network is functioning and it puts the system in an error state that it cannot recover from without unloading the modules. Is that possible? Shouldn't a new mount request attempt to make the communication to the servers again instead of just erroring out because there was an error previously?</p> <p>You should use the _netdev option, and thus avoid Lustre client mount attempts prior to network startup. The explanation is simple: you are trying<br/> to mount a network file system before you have a live network. I am not sure why you would need a module unload, that should not be necessary. Simply waiting for the net to be up should be enough. </p> <p>Dennis</p> <p>Any further questions or can we close this ticket?</p> <p>Thanks</p> <p>Peter</p> <p>IMHO this is still a bug. Yes, the _netdev option can help. However, the lustre client <b>should</b> gracefully handle problems when it tries to mount prior to the IB net being fully up, and a remount <b>should</b> be sufficient. The need for lustre_rmmod is not intuitive to systems admins. It can even be problematic if the client has another (active) Lustre mount and it is therefore impossible to unload the lustre modules.</p> <p>Thanks,<br/> Nathan</p> <p>Some notes here:</p> <ul class="alternate" type="square"> <li>I agree with Cliff that using _netdev can avoid this problem in most cases, which is what I use at home, but it isn't totally clear whether the IB network startup is treated the same as ethernet or not (i.e. ensure that the "mount _netdev filesystems" step will be appropriately delayed until after IB is up).</li> <li>I agree with Nathan that this is still a problem, since there can be other network problems that result in this "sticky" error (even with TCP), and it should be addressed.</li> <li>the "mount.lustre" command allows a "retry=N" mount option that will allow the client to repeat the mount up to N times on failure, but this may not be enough in this case, and can potentially hang the rest of the startup process.</li> </ul> <p>I believe that the root of the problem is with the ptlrpc module, since it starts the network connections when loaded, and may not retry establishing those connections if the network device was originally unavailable when it started.</p> <p>Close old ticket.</p> Development End date Thu, 7 Jun 2012 09:45:38 +0000 Rank 1|hzw0cn: Rank (Obsolete) 10136 Severity Start date Wed, 11 Apr 2012 09:45:38 +0000