https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8750 Lustre <p>when we mount 2 or more lustre fs on a client, the obd_timeout is the max of the all server obd_timeout. in some cases, it could be have some server evict due to that one of server does not wait obd_ping request enough time</p> <p>in my case, I have 2 lustre fs, Servers 2.5.X and some Clients 2.7, the first server have obd_timeout=100 and the second server have obd_timeout=300 so the obd_timeout inherited on the client is obd_timeout=300. the client send one obd-ping request each 75 seconds if just one obd_ping request is lost, the client could be evict, so it could be better to have a obd_timeout by filesystems or the min of the each servers filesystems</p> LU-8750

Wrong obd_timeout on the client when we have 2 or more lustre fs

Improvement Minor Resolved Duplicate Hongchao Zhang Antoine Percher Mon, 24 Oct 2016 09:39:30 +0000 Tue, 18 Apr 2023 17:24:35 +0000 Wed, 29 Mar 2023 22:32:08 +0000 Lustre 2.7.0 0 5 <p>I agree that this is a potential issue, and having a single global obd_timeout value is something that doesn't align with configurations where e.g. one filesystem is local and another is remote, and they should really have different timeout values.</p> <p>There are a few options that can be tried to resolve this problem without needing to wait for a patch and new release:<br/> 1) Try mounting the filesystems on a test client in the opposite order: the filesystem with the longer timeout (FS300) mounted first and the shorter timeout (FS100) mounted second, and then check <tt>lctl get_param timeout</tt> to see if this client uses the 100s timeout. If yes, then this could be put into production immediately without any further changes, except in the rare case where one filesystem is being mounted inside the other. If the client still has a timeout of 300s, then it appears that FS100 is using the default obd_timeout of 100s and not explicitly setting a timeout at all, and something more needs to be done.<br/> 2) As with #1 above, change the mount order to mount FS300 first and FS100 second, and also explicitly set the timeout parameter for FSshort via <tt>lctl conf_param &lt;fsname&gt;.sys.timeout=100</tt> and see if this allows the client to store the shorter timeout.<br/> 3) Set the timeout for FS100 to 300s to match FS300, so that the servers will wait up to 300s for the pings to arrive. However, this will also increase the recovery time for FS100 and that may not be desirable for some configurations.</p> <p>There are also potential code fixes for this problem, in particular we discussed to add a per-target <tt>ping_interval</tt> tunable in /proc, similar to <tt>max_rpcs_in_flight</tt> and <tt>max_pages_per_rpc</tt> that allows setting the ping interval for a single filesystem explicitly.</p> <p>Hi Hongchao,</p> <p>Can you please look into the suggested code fixes that Andreas has highlighted in the last comment?</p> <p>Thanks.<br/> Joe</p> <p>test output:<br/> 1) mount client with timeout 300 first, mount client with timeout 100 second<br/> the timeout is 100</p> <p>After setting timeout of FS100 to 300 explicitly, the timeout will be changed to 300 <tt>lctl conf_param FS100.sys.timeout=300</tt></p> <p>2) mount client with timeout 100 first, mount client with timeout 300 second<br/> the timeout is 300</p> <p>After setting timeout of FS300 to 100 explicitly, the timeout will be changed to 100 <tt>lctl conf_param FS300.sys.timeout=100</tt></p> <p>To properly fix this problem, it would be good to store the ping_interval and obd_timeout on a per-import basis. That would allow a single client to mount two or more different filesystems with different server timeouts (which the client can't control).</p> <p>With the newer userspace-driven parameter parsing (an upcall via udev to <tt>lctl</tt>) it <em>may</em> be easier to implement per-OBD timeouts relatively easily. By default, new OBD devices will inherit the global timeout value when they are created (stored in each obd_device or obd_export separately, and always used from the local device instead of the global value). If there is a <tt>timeout</tt> parameter in the configuration logs (which would normally generate an "<tt>lctl set_param timeout=&lt;value&gt;</tt>" upcall), this would be replaced by "<tt>&#42;.&lt;fsname&gt;-&#42;.timeout</tt>" so that the upcall for that filesystem's configuration log will only change the devices for the named filesystem.</p> <p>Closing this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9912" title="fix multiple client mounts with different server timeouts" class="issue-link" data-issue-key="LU-9912">LU-9912</a>, I've copied CC's over already.</p> Related LU-9912 LU-8066 LU-15246 LU-16749 Development Epic/Theme lnet Rank 1|hzyt5j: Rank (Obsolete) 9223372036854775807