https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5715 Lustre <p>Hi,</p> <p>The reboot gets hanged with lustre 2.5.2 and RHEL 6.5. If i unload the lustre modules using lustre_rmmod before reboot, it works. Appreciate your help here.</p> <p>my lustre.conf is as follows.</p> <p>options lnet networks="o2ib0(ib0)"</p> RHEL 6.5, MLNX_OFED_LINUX-2.2-1.0.1, CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz; Memory: 64GB; Kernel: 2.6.32-431.17.1.el6_lustre.x86_64; Lustre: 2.5.2-2.6.32_431.17.1 LU-5715

Reboot hangs due to lustre modules

Bug Major Resolved Duplicate WC Triage Chakravarthy N Tue, 7 Oct 2014 21:41:45 +0000 Sat, 29 Jan 2022 08:30:44 +0000 Sat, 29 Jan 2022 08:30:44 +0000 Lustre 2.5.2 1 9 <p>Are there any messages in kernel logs?</p> <p>Also is this something you started to experience recently and was all fine with older versions?</p> <p>There are no messages found in the syslog and everthing was fine until older versions. I dd not face the same issue with RHEL-6.4+Lustre-2.5 or RHEL6.4+Lustre-2.4</p> <p>I have seen this symptom in CentOS 6.3 with Lustre 2.1.4. (I don't have a newer configuration installed to try.) Lustre is set up using o2ib. The clients and all Lustre nodes have IPoIB enabled plus an Ethernet connection. The clients are generally busy full-time, which is to say that when a client shutdown is initiated it is likely that at least some processes have a Lustre directory or file opened (current working directory of a process, if nothing else).</p> <p>When the client hangs, there is no overt explanation - nothing in the syslog, etc. However, using IPMI to watch the client's virtual console showed that "/etc/init.d/rdma stop" was where the shutdown would hang. It would print that it was "Unloading OpenIB kernel modules" but it could not succeed because one (or more? I forget) of the OpenIB modules was in use. It would hang at that point.</p> <p>As a hack, it usually prevents the hanging if we change /etc/init.d/rdma so it calls lustre_rmmod; specifically, so that the original line "stop()" becomes</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>/etc/init.d/rdma hack</b></div><div class="codeContent panelContent"> <pre class="code-java">stop() { [ -x /usr/sbin/lustre_rmmod ] &amp;&amp; /usr/sbin/lustre_rmmod; real_stop } real_stop() </pre> </div></div> <p>This may or may not be related, but since it may be the symmetric issue at startup I'll mention it. On these clients, mounting the filesystem inside /etc/fstab using</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>/etc/fstab</b></div><div class="codeContent panelContent"> <pre class="code-java">&lt;IPoIB address&gt;@o2ib0:/lustre /mnt/lustre lustre defaults,_netdev 0 0 </pre> </div></div> <p>also generally fails: the system thinks the conditions for "_netdev" have been satisfied before ib0 is active so the mount fails. Stalling the mount one way or another is needed. (Do it explicitly later in the boot, or modify /etc/init.d/netfs so the check for _netdev waits for ib0, etc.) </p> <p>I can confirm this is still an issue using RedHat kernel with Lustre client:</p> <p>Linux 2.6.32-573.7.1.el6.x86_64 #1 SMP Thu Sep 10 13:42:16 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux</p> <p>Using standard RedHat Infiniband Support package.</p> <p>Hangs at Unloading IB Modules even though /usr/sbin/lustre_rmmod is being called prior. </p> <p>I can confirm the same issue here with RHEL6.8 and lustre 2.5.3, also using the RedHat Infiniband packages.</p> <p>This issue is resolved with patches for <a href="https://jira.whamcloud.com/browse/LU-8293" title="lnet init.d script missing insserv header" class="issue-link" data-issue-key="LU-8293"><del>LU-8293</del></a></p> Duplicate LU-8293 Development Epic server Rank 1|hzwy1r: Rank (Obsolete) 16029 Severity