https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9912 Lustre <p>If a single client is mounting multiple filesystems, but the servers are configured with different timeouts (e.g. local vs. WAN mounts) then the client will use the timeout value of the most recent filesystem mount. Since the <tt>obd_timeout</tt> value is global across all connections, if the most recent mount has a significantly higher timeout than the other filesystems, the client will not ping the servers often enough, resulting in client evictions during periods of inactivity.</p> <p>The timeout parameter is currently implemented as a single value for all filesystems currently mounted, but it seems possible to potentially fix this to work better when filesystems with multiple timeouts are mounted. One possibility is to use the minimum timeout set between the multiple filesystem configurations as the client ping interval. Currently, since <a href="http://review.whamcloud.com/2881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2881</a> the timeout is the maximum value from all different filesystems, which was to fix a problem with timeouts on the MGS. For clients it makes sense to use the minimum timeout between all filesystems mounted.</p> <p>Of course, it would be even better if the timeout value in the config was stored on a per-mountpoint or per-import basis, so that it is possible to have different timeouts for local and remote filesystems, or large and small filesystems, or interactive and batch filesystems, etc. I don't know how easily that would be implemented, but it would be the better long-term solution than a single timeout for the whole client.</p> LU-9912

fix multiple client mounts with different server timeouts

Bug Minor Open Unresolved Feng Lei Andreas Dilger Thu, 24 Aug 2017 17:43:14 +0000 Mon, 8 Jan 2024 23:35:29 +0000 Lustre 2.16.0 0 9 <p>The most logical place to me to expose this to admins is /sys/fs/lustre/mgc/MGC10.37.248.196@o2ib1/*. We could do a LND type approach. We treat the module parameters as global settings that by default applied to all network interfaces. Admin can configure via DLC each network interface and over ride the network interface values.</p> <p>Isn't obd_timeout controlled by adaptive timeouts ?</p> <p>Not exactly. Adaptive timeouts keep per-connection values for RPC processing and resends, but <tt>obd_timeout</tt> is still used for some case if there is no activity or are not directly related to RPC timeouts (e.g. pings when client is idle). </p> <p>For this ticket, we don't need to change the server code, but we should track the ping interval on a per server basis. </p> <p>Does obd_timeout show up as ptlrpc timeout messages ? <br/> Any idea what opcodes are reported (eg. 38/MDS_CONNECT or 400/OBD_PING) ?</p> <p>Thanks.</p> <p>To make this work properly, I think we should introduce a new device-specific parameter "<tt>&#42;.&#42;.timeout</tt>" (and equivalent for at_min, timeout, and other global parameters we want to fix), and then allow setting it directly. If the global <tt>timeout</tt> parameter is sent from the MGS it should only be applied to devices configured by that MGS, possibly tracked by <tt>fsname</tt> and translated into a glob like <tt>&#42;.&#42;fsname&#42;.timeout</tt> or equivalent when it is applied.</p> <p>This needs a similar change as patch <a href="https://review.whamcloud.com/45598" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45598</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15246" title="Add per device adaptive timeout parameters" class="issue-link" data-issue-key="LU-15246"><del>LU-15246</del></a> ptlrpc: add per-device adaptive timeout parameters</tt>" that changes the global "<tt>timeout</tt>" parameter to be per-device "<tt>&#42;.&lt;fsname&gt;&#42;.timeout</tt>" parameters.</p> <p>Separately, a patch for the MGC to translate config log parameter global settings of <tt>timeout</tt>, <tt>at_min</tt>, <tt>at_max</tt>, <tt>at_history</tt>, <tt>ldlm_enqueue_min</tt> to <tt>&#42;.&lt;fsname&gt;&#42;.&lt;param&gt;</tt> so that they only apply to devices from that filesystem rather than for all filesystems mounted by the client.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> do you mean the var <tt>obd_timeout</tt> in <tt>class_obd.c</tt>?</p> <p>Yes, this is the global "<tt>timeout</tt>" parameter set for the node. That is actually more problematic than <tt>at_min</tt>, etc. because <tt>obd_timeout</tt> controls the client's <tt>OBD_PING</tt> interval when it is idle. If client has smaller <tt>obd_timeout</tt> than server, no significant harm is done and client is more noisy than needed. If client has much larger <tt>obd_timeout</tt> than server then it will repeatedly be evicted by the servers, because they think it is dead (no contact for minutes).</p> <p>Implementation should be the same as previous patch - add a wrapper for to access <tt>obd_timeout</tt> from the device if set, otherwise (by default for now) use the global <tt>obd_timeout</tt> value.</p> <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50519" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50519</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9912" title="fix multiple client mounts with different server timeouts" class="issue-link" data-issue-key="LU-9912">LU-9912</a> ptlrpc: add per-device timeout param<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f29a032a06507b980281d9921413800fdf7d3846</p> <p>Lustre uses obd_timeout mostly for</p> <p>1) recovery case soft timeout, hard timeout, stale client waiting etc.</p> <p>2) ping interval and pinger eviction timeout. With <a href="https://jira.whamcloud.com/browse/LU-16002" title="Ping evictor delayed client eviction for 3 ping interval more than defined" class="issue-link" data-issue-key="LU-16002">LU-16002</a> ping interval and eviction multiplier were separated from obd_timeout. I don't see a strong case to have dependency between ping and recovery. Also I think that pinger timeout eviction could be based on AT and not obd_timeout. And network latency and busy ptlrpc could be addressed by pinger evictor.</p> <p>3) obd_timeout should cover network timeout layers. Something like different network configuration with/without routers, link changing, network reconfiguration etc.  </p> <p>Other parts - request timeout, ldlm timeouts are using Adaptive Timeouts mostly.</p> <p>I see the idea of having different ping interval for different FS. But I think that adding obd_timeout to every device is not good for original idea. It is better to have something like per-mount/per-FS variables.</p> <p> </p> <p>I also thought about per-mount/per-filesystem timeouts, but this is also complex because the pinging is actually done for each OBD device import basis, so having the timeout be on the OBD device makes sense. When this parameter is documented, the parameter will be described as "<tt>&#42;.<em>FSNAME</em>-&#42;.timeout=TIMEOUT</tt>", so that it applies to all OBD devices of that filesystem uniformly.</p> <p>One final thing that needs to be done here to fix the problem of a client mounting two different filesystems is to filter the "global" tunable parameters (<tt>timeout</tt>, <tt>ldlm_timeout</tt>, <tt>at_min</tt>, <tt>at_max</tt>, etc.) to only apply to OBD devices for that mountpoint. For example, if a client is mounting <tt>testfs1</tt>, then it should internally map <tt>timeout</tt> and related parameters to <tt>&#42;.testfs1&#45;&#42;.timeout</tt>. When mounting <tt>otherfs</tt> from a different MGS it should map <tt>timeout</tt> to <tt>&#42;.otherfs&#45;&#42;.timeout</tt>, etc. so that there is no confusion about global parameters affecting a different filesystem than intended.</p> Related LU-16749 LU-16002 LU-8750 LU-15246 Development Rank 1|hzziyv: Rank (Obsolete) 9223372036854775807 Severity