https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1014 Lustre <p>While testing Lustre master branch on our production system we ran into the issue of the timeout being incorrect on the MGS. This was causing the MGS to drop OSS and MDS connections. Also discovered if we changed the obd_timeout parameter anywhere to another value that value was never propagated to the rest of the system. </p> Any Lustre 2.X setup. LU-1014

MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system.

Bug Blocker Resolved Fixed Lai Siyao James A Simmons Fri, 20 Jan 2012 09:22:09 +0000 Fri, 24 Aug 2012 16:28:03 +0000 Fri, 24 Aug 2012 16:28:03 +0000 Lustre 2.2.0 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.3.0 Lustre 2.4.0 0 6 <p>Here is a easy example of seeing the problem. Building a test file system with --param sys.timeout=20 to every device and then mounted this file system. Doing the following command gives:</p> <p>pdsh -w spoon02,maid-mds1,barry-mds1,barry-oss<span class="error">&#91;1-4&#93;</span> "lctl get_param timeout"<br/> spoon02: timeout=20<br/> barry-oss3: timeout=20<br/> barry-oss4: timeout=20<br/> maid-mds1: timeout=100<br/> barry-oss2: timeout=20<br/> barry-oss1: timeout=20<br/> barry-mds1: timeout=20</p> <p>As you can see the timeout on the MGS unit is set to the default 100 meaning the --param-sys.timeout=20 flag is being ignored. </p> <p>Now for the technical reasons for why I think this is the case. Information such as the timeout is stored in the MGS logs via the function mgs_write_log_target. This information is passed with the struct mgs_target_info. This is all done when a target is registered with the MGS with mgs_handle_target_reg which is called when a RQF_MGS_TARGET_REG rpc is recieved. You will notice that the RQF_MGS_TARGET_REG is sent by mgc_target_register which happens when the key KEY_REGISTER_TARGET is received in mgc_set_info_async. This is all triggered by a mount of a target as you can see in obd_mount.c with the function server_register_target which is used to start the service targets i.e MDTs and OSTs. This is all to register via the MGC to the MGS for the specific targets. Well for the case of the MGS it has to be started BEFORE the MGC. Then the MGC is registered before all the server targets. This is the not the case for the MGS so no data is ever written to the logs of the MGS unit since the MGS is never registered to itself.</p> <p>Something that also may be of interest is what is stored in the CONFIGS section of the MGT:</p> <p><span class="error">&#91;root@maid-mds1 ~&#93;</span># tunefs.lustre --dryrun /dev/sda<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lustre<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: sys.timeout=20</p> <p> Permanent disk data:<br/> Target: MGS<br/> Index: unassigned<br/> Lustre FS: lustre<br/> Mount type: ldiskfs<br/> Flags: 0x74<br/> (MGS needs_index first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: sys.timeout=20</p> <p>exiting before disk write.</p> <p>Oleg</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Jason, tunefs.lustre dumps mountdata, not llog config, which is the data written by mkfs, not the one directly used in device startup. The problem is just as James pointed out, MGS doesn't follow normal target device setup process, but just get created by lustre_start_simple().</p> <p>Review is on <a href="http://review.whamcloud.com/#change,2139" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2139</a></p> <p>In this patch, MGS will process sys parameters from on-disk mountdata, that means, sys parameters can't be changed via `lctl set_param/conf_param ...` temporarily or permanently on disk, but through tunefs.lustre. IMO this is reasonable, because MGS may hold config for several setups, sys parameters change on one setup should not affect other setups.</p> <p>James, would you verify the patch works?</p> <p>James</p> <p>When do you expect to be able to verify whether this patch works?</p> <p>Thanks</p> <p>Peter</p> <p>Peter/Lai :</p> <p>Taking away the lctl interface from a client can potentially break some legacy administrative scripts. For example our client startup script mounts the filesystem looks at the values in /proc/sys/lustre/</p> {timeout,ldlm_timeout,at_min,at_max} <p> and sets them if they are not correct. Now with this patch the timeout <b>should</b> always be correct, but will it still be queryable via lctl? Is it reasonable to expect that you need to unmount your MGT to change this value in production (I can see a use case there where it is not reasonable)? What affects would it have on the currently mounted MDT's and OST's - would they then need to be unmounted and re-mounted to pick up the change? My concern here is availibility and how we have architected the Spider filesystems here at ORNL (common MGS for all filesystems). Same concern for currently mounted clients - or would recovery handle getting the timeout value set on the client? Are there any precedents for removing a tunable from lustre in terms of compatibility that are of concern?</p> <p>Thanks,<br/> &#8211;<br/> -Jason</p> <p>I plan to test it tomorrow and do regression testing over the weekend. Monday we will boot our test cray system in the latest Lustre version to see if it holds up.</p> <p>Okay I have tested your patch. Test the MGS formated with --param sys.timeout and also tried lctl <span class="error">&#91;s/g&#93;</span>et_param timeout<span class="error">&#91;=XXX&#93;</span> on clients and servers. On the MGS lct get_param timeout reported the right value when compared to the tune.lustre reported values which were passed in by the format command. I also wanted to see if you could change the timeout with lctl and I discovered you can change the value on any client or server including the MGS. That changed value also didn't change back to the old value at any time as long as you didn't reboot the node. The regression I did see is if you did change the obd_timeout on any client or server that new value did not propagate to all the nodes of the file system.</p> <p>Ahh, I misunderstood it, I used to think that timeout can only be changed through mountconf, but actually `lctl set_param timeout=xxx` just write to /proc/sys/lustre/timeout, which corresponds to a global variable 'obd_timeout', and it won't be propagated to the file system; while `lctl conf_param &lt;target&gt;.sys.timeout=xxx` will trigger mountconf update on the target.</p> <p>Could you confirm it is regression? or do I miss anything?</p> <p>Before I do a lctl conf_param lustre.sys.timeout=150 on the MGS (maid-mds1, barry-mds is the MDS)</p> <p>pdsh -w maid-mds1,barry-mds1,barry-oss<span class="error">&#91;1-4&#93;</span>,spoon01 "lctl get_param timeout"<br/> barry-mds1: timeout=300<br/> barry-oss2: timeout=300<br/> barry-oss4: timeout=300<br/> barry-oss1: timeout=300<br/> barry-oss3: timeout=300<br/> maid-mds1: timeout=300<br/> spoon01: timeout=300 (client)</p> <p>After a pdsh -w maid-mds1 "lctl conf_param lustre.sys.timeout=150"</p> <p>pdsh -w maid-mds1,barry-mds1,barry-oss<span class="error">&#91;1-4&#93;</span>,spoon01 "lctl get_param timeout"<br/> barry-oss1: timeout=150<br/> barry-oss4: timeout=150<br/> barry-mds1: timeout=150<br/> barry-oss3: timeout=150<br/> barry-oss2: timeout=150<br/> maid-mds1: timeout=300<br/> spoon01: timeout=300</p> <p>As you can see the client and the MGS are not updated with the new obd_timeout value.</p> <p>James, I did the same test, but the result is different: client always updates timeout, but MGS never. And the result is identical on both patched and unpatched code.</p> <p>The code shows that although MGS has a MGC, but it doesn't enqueue config lock, so when config changes, it's not notified. I'll update the patch to fix it later.</p> <p>Could you verify that client not update its timeout?</p> <p>I'm a idiot. The client had the LNET set up but I didn't have the file system mounted. Once I mounted a file system on the client I seen the timeout change on the client. Only the MGS timeout doesn't change which it appears that you know what the solution is. Look forward to your new patch.</p> <p>Also could you update cfg/local.sh from</p> <p>MGS_MKFS_OPTS="--mgs --device-size=$MGSSIZE"</p> <p>to </p> <p>MGS_MKFS_OPTS="--mgs --device-size=$MGSSIZE --param sys.timeout=$TIMEOUT"</p> <p>Looking at the the mgs parameter setting I'm seeing mgs_iocontrol -&gt; mgs_setparam -&gt; which does mgs_write_log_param as well as call mgs_revoke_lock. Mgs_revoke_lock does a local "ldlm_cli_enqueue_local" which handle the config changes. It should be notified which in turn calls mgs_completion_ast_config</p> <p>James, as I've said, although standalone MGS has a mgc device, because MGS doesn't have config of its own, this mgc doesn't enqueue any config lock, so upon config changes, standalone MGS won't be notified.</p> <p>In the new patch I'll commit later, the mgc on MGS will enqueue '&lt;profile&gt;-params' config lock, this config is global parameters for the system. But there is one thing strange: even when you issue `lfs conf_param &lt;profile&gt;-MDT0000.sys.timeout=xxx` (or any other target/client device), all the config files will update its timeout value (ie. &lt;profile&gt;-params and &lt;profile&gt;-MDT0000, &lt;profile&gt;-OSTxxxx and &lt;profile&gt;-client), I'm not clear whether this is intended. So with this patch, there is no need to change cfg/local.sh script, because the timeout value will be set by the last device timeout parameter.</p> <p>Thanks for the patch and info. I'm just attempting to understand the lustre code in greater depth. With the second patch I now understand how config is set up. Is patch 3 a total replacement of patch 2 or do you need to use both? I applied patch number 2 and patch number 3 and tested them with success. The timeout format parameter on the MGS is recognized and when I do a lctl conf_param it updates the timeouts correctly. As for was the syncing of obd_timeout intended, well I know our site is depended on that being available.</p> <p>Patch 3 is a total replacement for patch 2. The original issue is that MGS doesn't have llog config (normally for target like MDT/OST this config is created at device setup time, which will connect to MGS and register itself), so that it doesn't enqueue any config lock.</p> <p>In patch 2 I tried to parse sys parameters through mountdata created by mkfs.lustre in MGS startup, but in this way MGS won't be notified if config changes; so in patch 3 MGC on MGS will enqueue '&lt;profile&gt;-params' config lock, this config contains global sys parameters, therefore MGS can process global sys parameters like other nodes process their config. Because patch 2 is a special way to process parameters, and patch 3 is using a normal method to process config, and can cover the functionality of patch 2, I'd like use patch 3 only.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Landed for 2.2</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=ofa/494/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,ofa #494</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/mgc/mgc_request.c</li> <li>lustre/obdclass/obd_mount.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1014" title="MGS with sys.timeout is ignored and if sys.timeout is changed its not synced across the file system." class="issue-link" data-issue-key="LU-1014"><del>LU-1014</del></a> mountconf: MGS should process parameter config (Revision 6cde8692384a1d3a6d0e8114629fa943c951fe40)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=6cde8692384a1d3a6d0e8114629fa943c951fe40" class="external-link" target="_blank" rel="nofollow noopener">6cde8692384a1d3a6d0e8114629fa943c951fe40</a><br/> Files : </p> <ul> <li>lustre/obdclass/obd_mount.c</li> <li>lustre/mgc/mgc_request.c</li> </ul> <p><a href="http://review.whamcloud.com/#change,2667" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2667</a><br/> back port to b2_1</p> <p>There is some concern that this patch is causing breakage on the Orion branch, and may be fundamentally incorrect, per comments in ORI-651. Since this problem is longstanding and has never been reported in production, I would prefer that this patch be reverted from b2_1 until there is a more thorough understanding of the issue. </p> <p>I believe we need to reopen this bug and revisit this issue. After this patch was applied I am now encountering the following issue which causes the MGS to crash when mounted, ORI-651.</p> <p>Now leaving aside the bag paging request which is clearly just a a use-after-free bug in the initial patch. The more that I started looking at this the more I'm convinced this patch needs to be reverted and reworked. I say that because the current implementation is at odds with my understanding of the Lustre MGS architecture (correct me if I'm wrong about this).</p> <p>But this patch expects the MGS service name to include the filesystem name (&lt;fsname&gt;-MGS). However, until the Lustre clients support connecting to multiple MGSs that's impossible since the MGS is a resource which is shared between multiple filesystems. This change needs to be reworked to cleanly handle the multiple filesystem case.</p> <p>Also note that reverting <a href="http://review.whamcloud.com/2139" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2139</a> from orion resolves the crash seen in ORI-651.</p> <p>Please refer to ORI-662 instead of ORI-651. The key issue is separate MGS setup, which has no ldd_fsname, so param log has name '-params' and things are not working as expected.</p> <p>ORI-651 has normal names and the only issue there is memory access which looks like Orion-side issue.</p> <p>No, memory issue is not Orion-related, there is clear use-after-free case in original patch, check 'logname' is used being freed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ OBD_FREE(logname, MGS_PARAM_MAXLEN); + <span class="code-keyword">if</span> (rc) { + CERROR(<span class="code-quote">"failed to process parameters %s: %d\n"</span>, + logname, rc); + GOTO(out_mnt, rc); </pre> </div></div> <p>That causing issues we've seen, and there is still issue about separate MGS case with no fsname</p> <p>Another question about this patch is about fsname-params log processing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (IS_MGS(lsi-&gt;lsi_ldd) &amp;&amp; + !(lsi-&gt;lsi_lmd-&gt;lmd_flags &amp; LMD_FLG_NOMGS)){ + struct config_llog_instance cfg; + <span class="code-object">char</span> *logname; + + OBD_ALLOC(logname, MGS_PARAM_MAXLEN); + <span class="code-keyword">if</span> (logname == NULL) + GOTO(out_mnt, rc = -ENOMEM); + strcpy(logname, lsi-&gt;lsi_ldd-&gt;ldd_fsname); + strcat(logname, <span class="code-quote">"-params"</span>); + + memset(&amp;cfg, 0, sizeof(cfg)); + rc = lustre_process_log(sb, logname, &amp;cfg); + OBD_FREE(logname, MGS_PARAM_MAXLEN); + <span class="code-keyword">if</span> (rc) { + CERROR(<span class="code-quote">"failed to process parameters %s: %d\n"</span>, + logname, rc); + GOTO(out_mnt, rc); + } </pre> </div></div> <p>so it is processed only on MGS, second condition should be always true as well, because MGS cannot be with NOMGS flag I suppose, at least that sounds strange. But we don't need params processing on MGS, it should be processed on every node except MGS, right?</p> <p>I didn't notice that during inspection, sorry.</p> <p>I believe the flag is needed as well because IS_MGS(..) could equal IS_MDS(..). I discovered this bug when the MGS and MDS were separate servers thus they didn't share the same llogs. Yes we do need params processing so that when for example the timeout is changed on the system it gets set on the MGS as well.</p> <p>This is actually a MGS design issue: MGS allows to be shared by multiple filesystems, and it will create a &lt;profile&gt;-params for each filesystem, besides, MGS will create a lustre-params by default (the name 'lustre' is written into mountdata by mkfs.lustre by default, and that should be why ORION code can't find this llog config). In my previous patch MGC on MGS will enqueue lustre-params config lock, but sys.timeout won't be synced if filesystem name is not 'lustre'. And I don't know how this can work if MGS holds llog configs of several filesystems with different sys.timeout.</p> <p>Any comment is welcome!</p> <p>Some notes:</p> <ul class="alternate" type="square"> <li>it probably would make sense to remove "lustre" as the default filesystem name. The admin should specify a proper filesystem name for any filesystem, and mkfs.lustre should fail if it is not specified, along with updating man pages and usage() output. We've already had reports of a site having two filesystems with the same name. That should be done in a separate bug.</li> <li>There should not be any harm to have a larger timeout on the MGS than the rest of the filesystem, since at worst the clients will be pinging the MGS more frequently than it expects. In this regard, the MGS could check all of the configuration files, and only <em>increase</em> the timeout to the maximum one found in <em>any</em> config log.</li> <li>A more important question is to understand what the MGS uses timeouts for at all? Is this only to evict stale clients? I suspect the reason we haven't seen this problem very much is that the timeout in the config log is rarely changed to be more than 4x the timeout (the server allows clients to miss 4 pings before evicting). If this is just the timeout, it might make sense to have the MGS just track the average ping interval from the client, and only evict the client if it misses 4x the average ping interval? The benefit is that the MGS doesn't need to parse the different obd_timeouts at all. The drawback is that this is extra code and runtime overhead and it allows stale clients to linger a bit longer, though if there are DLM callback timeouts the client will still be evicted.</li> </ul> <p>&gt; I believe the flag is needed as well because IS_MGS(..) could equal IS_MDS(..).</p> <p>James, IS_MGS() checks just LDD_F_SV_TYPE_MGS flag is set, you are right, it is true for combined MGS+MDT but in that case there can't be LMD_FLG_NOMGS, because there is MGS combined with MDT. Another issue with patch is 'else if', as I can see that check will be just skipped if first condition was true.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">1828 <span class="code-keyword">if</span> (!(lsi-&gt;lsi_lmd-&gt;lmd_flags &amp; LMD_FLG_NOSVC) &amp;&amp; 1829 (IS_OST(lsi-&gt;lsi_ldd) || IS_MDT(lsi-&gt;lsi_ldd))) { 1830 rc = server_start_targets(sb, mnt); </pre> </div></div> <p>So with combined MDT+MGS this check is true and MDT service will be started but 'else if' will not be even checked, so that code under 'else if' will work only for separate MGS. But I thought we need params processing on other nodes too?</p> <p>Also I don't agree we have to change MGS parameters along with filesystem params, just because MGS is shared. Probably MGS params should be set separately and stored somewhere in mgs-params config which contains just MGS params?</p> <p>New patch at <a href="http://review.whamcloud.com/#change,2843" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2843</a></p> <p>I just tested the new patch. No regressions, and worked properly as expected on my end. Thank you. Now to see if ORI 651 still crashes for Livermore.</p> <p>The patch behaves as expected except for one thing which appears to be a lctl bug. I logged into the MGS to make sure it is not a pdsh issue.</p> <p><span class="error">&#91;root@tick-mgs1 ~&#93;</span># lctl get_param timeout<br/> timeout=150<br/> <span class="error">&#91;root@tick-mgs1 ~&#93;</span># lctl conf_param lustre.sys.timeout=70 <br/> <span class="error">&#91;root@tick-mgs1 ~&#93;</span># lctl get_param timeout<br/> timeout=150<br/> <span class="error">&#91;root@tick-mgs1 ~&#93;</span># lctl conf_param lustre.sys.timeout=70<br/> <span class="error">&#91;root@tick-mgs1 ~&#93;</span># lctl get_param timeout<br/> timeout=150</p> <p><span class="error">&#91;root@tick-mgs1 lustre&#93;</span># cat /proc/sys/lustre/timeout <br/> 150<br/> <span class="error">&#91;root@tick-mgs1 lustre&#93;</span># echo "70" &gt; /proc/sys/lustre/timeout <br/> <span class="error">&#91;root@tick-mgs1 lustre&#93;</span># cat /proc/sys/lustre/timeout <br/> 70<br/> <span class="error">&#91;root@tick-mgs1 lustre&#93;</span># lctl get_param timeout<br/> timeout=70</p> <p>As you can see something strange is going on with the lctl command. Also the first time you change the value with lctl it appears to work but after that it doesn't anymore.</p> <p>That is because the previous timeout parameter to set the value to 150 is still in the config log, and sets the "obd_timeout_set" switch, and later entries in the log with lower values are ignored. Since it is not typical to twiddle with the MGS timeouts, you can still use "lctl set_param timeout=70" on the MGS (instead of "lctl conf_param") and it should still temporarily override the MGS obd_timeout value it got from the config log.</p> <p>I have been using the patch at <a href="http://review.whamcloud.com/#change,2881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2881</a> with great success for some time.</p> <p>Move to 2.3/2.4, since the patch landed to 2.1/2.2 was later reverted.</p> <p>Landed for 2.3 and 2.4</p> Related Development Rank 1|hzv5tr: Rank (Obsolete) 4478 Severity