Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-961] lfs df -h hangs https://jira.whamcloud.com/browse/LU-961 Lustre <p>When I try to use <tt>lfs df -h</tt> the command hangs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 767.8M 35.1M 681.5M 5% /mnt/lustre[MDT:0] </pre> </div></div> <p>Any ideas why this is hanging?</p> TCP network LU-961 lfs df -h hangs Bug Minor Closed Fixed Brian Murrell Brian Murrell Wed, 4 Jan 2012 12:06:39 +0000 Thu, 5 Jan 2012 13:14:07 +0000 Thu, 5 Jan 2012 13:13:52 +0000 Lustre 2.1.0 0 3 <p>Can we get message logs from the MDS and OSS(s)?</p> <p>I have way too many OSSes to add the logs from all of them. Can you be more specific about which OSSes you want logs from?</p> <p>How about the MDS and the OSS that has OST0000?</p> <p>Find them attached.</p> <p>I am just in the process of rebooting my whole cluster to see if the problem is resolved by doing that.</p> <p>After rebooting things still don't look good:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lfs df -h UUID                       bytes        Used   Available Use% Mounted on lustre-MDT0000_UUID       767.8M       35.1M      681.5M   5% /mnt/lustre[MDT:0] OST0000             : inactive device lustre-OST0001_UUID      1007.9M       52.2M      904.5M   5% /mnt/lustre[OST:1] filesystem summary:      1007.9M       52.2M      904.5M   5% /mnt/lustre </pre> </div></div> <p>I am seeing the following network related errors in both the MDS and the OSS logs. Can you confirm that your LNET network is up and functioning as expected? Particularly i would like to see if the OSS that you provided logs from can ping the MDS over LNET (i.e. lctl ping &lt;ipofMDS&gt;), and reversely can the MDS ping the OSS?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jan 4 10:18:35 mds1 kernel: LustreError: 4047:0:(socklnd.c:2420:ksocknal_base_startup()) Can't spawn socknal scheduler[0]: -513 Jan 4 10:18:35 mds1 kernel: LustreError: 105-4: Error -100 starting up LNI tcp Jan 4 10:18:35 mds1 kernel: LustreError: 4047:0:(events.c:728:ptlrpc_init_portals()) network initialisation failed </pre> </div></div> <p>Hi,</p> <p>It seems that communication between oss1 and mds2 is working:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@oss1 ~]# lctl ping mds2 12345-0@lo 12345-192.168.122.155@tcp [root@mds2 ~]# lctl ping oss1 12345-0@lo 12345-192.168.122.147@tcp </pre> </div></div> <p>I'm noticing the time on the messages that you are referring to. I must apologize for not mentioning when I attached the logs for not mentioning that the cluster was down for maintenance and testing at that point. It wasn't up for production until a few hours later than that so probably looking further in the log might be helpful.</p> <p>Can you please upload the full MDS and OSS logs following the reboot? It appears the MDS was still coming up from the reboot when the log was captured.</p> <p>Thanks!</p> <p>I've attached the full message log from mds2 which is actually the MDS on this filesystem. I erroneously gave you mds1's log yesterday. mds1 is the current MGS.</p> <p>I've also attached the newer oss1 logs from the point where the previous oss1 log attachment ended.</p> <p>Could we get logs from one of the clients experiencing the hang? Also, is the hang experienced from all clients, only one, or only a certain sub-set of clients within a particular portion of the network?</p> <p>Here are the client logs</p> <p>Faulty network discovered.</p> Development Rank 1|hzvhl3: Rank (Obsolete) 6498 Severity