https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1344 Lustre <p>Customer reported a number of clients were evicted. All clients had difficulties communicating with OSTs on a single OSS. Johann has looked at the client logs but I did not have server logs at the time. I now have the server logs. I have attached them to this ticket. I need recommendations on how to prevent this from happening in the future.</p> <p>Should I consider changing the OBD timeout from the default 100s?</p> <p>Should I consider reducing the number of OST service threads (default 256)?</p> CentOS 5.5 on Lustre servers<br/> RHEL 6.1 on clients LU-1344

Evicted Clients

Bug Major Resolved Cannot Reproduce Liang Zhen Dennis Nelson Thu, 26 Apr 2012 12:43:09 +0000 Fri, 19 Oct 2012 10:45:47 +0000 Fri, 19 Oct 2012 10:45:47 +0000 Lustre 1.8.6 0 3 <p>The server logs confirm that there are network problems.<br/> Liang (our lnet expert) is having another look at the logs just in case.</p> <p>obd timeout is now automatically adjusted, there should be no need to change this. <br/> You can check for the file 'timeouts' - there is one for various services under /proc/fs/lustre. <br/> That file provides a history and will show if there are timeout issues. </p> <p>The errors appear to show a network/other failure rather than a server overload. The client evictions<br/> track to errors on the server. <br/> There are only LustreErrors in the server logs. Are there any indications of a network failure?<br/> What was the load on the server when the clients dropped connections? <br/> I would suggest upgrading to Lustre 1.8.7 as there are improvements in that release. </p> <p>So, I understand that obd timeout mostley deprcated with the introduction of adaptive timeouts. That is the reason it is still set to the default. A coworker pointed out the following passage from the manual and suggested that it might be helpful to increase it:</p> <p>In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout) value was used as the maximum completion time for all RPCs; this value also affected the client-server ping interval and initial recovery timer. Now, with adaptive timeouts, obd_timeout is only used for the ping interval and initial recovery estimate. When a client reconnects during recovery, the server uses the client's timeout value to reset the recovery wait period; i.e., the server learns how long the client had been willing to wait, and takes this into account when adjusting the recovery period.</p> <p>I found that there is some sar data being collected and cpu idle time never went below 90%. The reason I am thinking that this might be load related is based on the following server entry:</p> <p>Apr 20 20:05:14 lfs-oss-1-13 kernel: Lustre: Service thread pid 32236 completed after 278.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).</p> <p>Also, the clients did not have any issues with OSTs on any of the other servers during this time.</p> <p>All kernel errors are directed to /var/log/kern.log. You got that entire file. /var/log/messages is empty of any messages at all after the last boot 12 days ago. I have not seen any logs on the server that indicate there was a network issue at the time.</p> <p>Liang is reviewing the logs</p> <p>Any update on the review of the logs?</p> <p>We haven't seen this since, we can close it. </p> Development Rank 1|hzv39z: Rank (Obsolete) 4030 Severity