https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-828 Lustre <p>Hi ,</p> <p>We have lustre 2.0 at our setup with 2 mds servers and 6 oss servers.</p> <p>Facing an issue where "lfs check servers" output varies from node to node.</p> <p>Please find below the outputs of two different clients took at the same time.</p> <p>Request your help in solving this issue.</p> <p>Also find attached var log messags of the node having this error.</p> <p><span class="error">&#91;root &lt;at&gt; cn367 ~&#93;</span># lfs check servers</p> <p>scratch-OST0011-osc-ffff810c01a9c000 active.</p> <p><span class="error">&#91;root &lt;at&gt; mgmt00 ~&#93;</span># lfs check servers</p> <p>error: check 'scratch-OST0011-osc-ffff810c056c4000' Resource temporarily unavailable</p> <p>Thanks &amp; Regards,<br/> N.Chakravarthy.</p> RHEL 5.5<br/> 2* MDS servers, 6*OSS servers handling 3OST each (HW config: Dual Intel Westmere processor with 6 cores each, 24GB Memory)<br/> no. of clients: 368 LU-828

Lustre Client Unstable

Task Critical Resolved Cannot Reproduce WC Triage Chakravarthy N o2iblnd Wed, 9 Nov 2011 00:38:55 +0000 Fri, 27 Jan 2012 18:41:05 +0000 Fri, 27 Jan 2012 18:41:05 +0000 Lustre 2.0.0 0 0 <p>Check the system logs for the two nodes, it is quite possible for one client to be happy and a second client to be unable to reach a particular server. This can be caused by network issues or other errors Check system logs and dmesg for both clients, and the OSS involved, there should be some addition errors that will help you sort out the situation. </p> <p>Thanks for your reply...</p> <p>I could see errors in logs as below... Does it means that it's due to mcast packet drops in infiniband network?</p> <p>ADDRCONF(NETDEV_UP): ib0: link is not ready<br/> ib0: enabling connected mode will cause multicast packet drops<br/> ib0: mtu &gt; 4092 will cause multicast packet drops.<br/> ib0: mtu &gt; 4092 will cause multicast packet drops.<br/> ib1: enabling connected mode will cause multicast packet drops<br/> ib1: mtu &gt; 4092 will cause multicast packet drops.<br/> ib1: mtu &gt; 4092 will cause multicast packet drops.</p> <p>LustreError: 11-0: an error occurred while communicating with 10.2.2.187@o2ib. The mds_getxattr operation failed with -95<br/> LustreError: 11-0: an error occurred while communicating with 10.2.2.187@o2ib. The mds_getxattr operation failed with -95<br/> LustreError: 11-0: an error occurred while communicating with 10.2.2.187@o2ib. The mds_getxattr operation failed with -95<br/> LustreError: 11-0: an error occurred while communicating with 10.2.2.187@o2ib. The mds_getxattr operation failed with -95<br/> LustreError: 11-0: an error occurred while communicating with 10.2.2.187@o2ib. The mds_getxattr operation failed with -95</p> <p>The only difference i could see is this node has two IB interfaces configured in load balancing mode, the lnet entry is "options lnet networks=o2ib0". Hope it's ok.</p> <p>Just wanted to update on this issue, whenever we do "ls" on the lustre fs then only it's showing this error. We've checked the same with the non problematic node as well the issue remains...</p> <p>Please suggest...</p> <p>Appreciate your help ASAP, since the entire production is on toss...</p> <p>Please do the needful.</p> <p>Sample compute node syslog messages where issue is noticed</p> <p>Syslog messages of lustre MGS,MDS,OSS Server nodes for /home lustre filesystem </p> <p>Syslog messages of lustre MGS,MDS,OSS Server nodes for /scratch lustre filesystem </p> <p>Hi Cliff,</p> <p>I have uploaded following 3 archive files having Syslog messages of client node where these problem is noticed, and MGS,MDS,OSS server nodes messages for both /home /scratch lustre filesystems. Could you please quickly glance through the logs to see what is the cause for the issue and unexpected symptoms (like Resource temporarily unavailable and ls,cat cmd hangs etc) reported here.</p> <p>cn363-computenode_messages.rar<br/> HOME_MESSAGES.rar<br/> SCRATCH_MESSAGES.rar</p> <p><span class="error">&#91;root &lt;at&gt; cn367 ~&#93;</span># lfs check servers<br/> scratch-OST0011-osc-ffff810c01a9c000 active.</p> <p><span class="error">&#91;root &lt;at&gt; mgmt00 ~&#93;</span># lfs check servers<br/> error: check 'scratch-OST0011-osc-ffff810c056c4000' Resource temporarily unavailable</p> <p>Additional symptoms are, <br/> When we try to ls dir /scratch/qgr/R8 dir it stucks. And resource become unavialbale.<br/> We tried with ls --color=none , in this case we are able to list the files.<br/> But when we do ls -l --color=none it again stucks.<br/> we are not able to cat the file on which ls -l stucks others are working fine.</p> <p>Thanks &amp; Regards<br/> -Raghu</p> <p>If your infiniband network is dropping packets, that would cause this issue. </p> <p>Cliff,</p> <p>Just an update on this issue...</p> <p>We've taken the downtime of the entire lustre and "ls, du" everthing started working..</p> <p>To my understanding the recovery, open files and caching in the client has solved the issue.</p> <p>Could you please suggest some permanant solution for this issue like clearing the cache, close open files automatically without doing lfsck?</p> <p>Appreciate your early help on this..</p> <p>Cliif,</p> <p>Appreciate your suggestions on this, since we are in a bad shape... Please do the needful.</p> <p>Since you are an unsupported customer, the only thing I can suggest is that you upgrade to the latest Lustre 2.1.0 release to determine if this is fixing your problem. To get Whamcloud support for your system, please contact info@whamcloud.com for more information.</p> Development Epic client Rank 1|hzw0rb: Rank (Obsolete) 10203