https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4939 Lustre <p>Is there a sane way to determine what NIDs a server is configured with as determined by the MGS?</p> <p>I know there's <tt>lctl list_nids</tt>, but that's not what I want. I need to determine which NIDs the MGS has for a given server, as I think the MGS is handing a client a non-existent NID for a server.</p> LU-4939

Need to be able to sanely query and change MGS configuration information

Bug Minor Resolved Fixed Niu Yawei Prakash Surya llnl patch Tue, 22 Apr 2014 01:21:57 +0000 Thu, 18 Jul 2019 19:25:16 +0000 Fri, 2 Nov 2018 10:52:58 +0000 Lustre 2.12.0 Lustre 2.10.7 0 12 <p>Is there a (sane) way to query the configuration stored on the MGS in general?</p> <p>Sigh.. hex editor it is???</p> <p>lol!</p> <p>Hi, Prakash. llog_reader can read the config file on MGS (mount MGS device as ldiskfs or dump out the config files by debugfs first), is this what you want?</p> <p>Would lctl get_param osc.*.import give you what you what? <br/> It lists the nids that are in use. See the 'connection' section.</p> <p> connection:<br/> failover_nids: <span class="error">&#91;192.168.120.13@o2ib&#93;</span><br/> current_connection: 192.168.120.13@o2ib</p> <p>Cliff, that might work, but I suspect he would like to validate the configuration on the MGS before he starts the filesystem with a potentially broken config.</p> <p>This client can't mount the MDS, so I'm not sure the <tt>lctl get_param</tt> will work.</p> <p>Just to give a little more info on my actual problem..</p> <p>From looking at the client lustre log, I see these two lines:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000080:18.0:1398123825.044168:0:18652:0:(obd_config.c:1079:class_process_config()) adding mapping from uuid 10.1.1.212@o2ib9 to nid 0x500090a0101d4 (10.1.1.212@o2ib9) 00000020:00000080:18.0:1398123825.044199:0:18652:0:(obd_config.c:1079:class_process_config()) adding mapping from uuid 10.1.1.212@o2ib9 to nid 0x200000a0101d4 (10.1.1.212@tcp) </pre> </div></div> <p>The MDS is at NID <tt>10.1.1.212@o2ib9</tt>, and NID <tt>10.1.1.212@tcp</tt> doesn't even exist. The client is on the <tt>tcp0</tt> network, and routed through to <tt>o2ib9</tt>. We can <tt>lctl ping</tt> from both: the client to the MDS, and the MDS to the client (so it doesn't look to be a routing issue).</p> <p>What I think is going on, is <em>somehow</em> (we don't know how) the MGS has "bad" data regarding the MDS nids. I <em>think</em> the MGS has the <tt>o2ib9</tt> nid and the <tt>tcp</tt> nid listed for the MDS. Thus when the <tt>tcp</tt> client tries to mount, I think this occurs:</p> <p> 1. it connects to the MGS just fine<br/> 2. get's the list of NIDs for the MDS (<tt>10.1.1.212o2ib9</tt>, and <tt>10.1.1.212@tcp</tt>)<br/> 3. decides the <tt>10.1.1.212@tcp</tt> NID is the "best" and tries to connect using that<br/> 4. connect to <tt>10.1.1.212@tcp</tt> fails because that NID doesn't exist</p> <p>Mounting from another client on another network (<tt>o2ib2</tt>) works just fine. My guess, is that's because the "good" client just happens to ignore the <tt>10.1.1.212@tcp</tt> NID and uses the valid NID of <tt>10.1.1.212@o2ib9</tt>.</p> <p>Granted, this is just from poking at the lustre logs on the "bad" client. I'd really like to dump the lustre MGS config data to know for sure, but obviously that's not easy to do. Sigh.</p> <p>We just did a <tt>--writeconf</tt> on the servers to reset the NIDs and that has resolved the problem. The <tt>tcp0</tt> client can mount the filesystem just fine now.</p> <p>Can I make a feature request for a tool to read the configuration information from the MGS? Perhaps the llog_reader is a good place to start (as Niu suggested), but mounting the MGS devices as ldiskfs doesn't seem like a "correct" solution. Proper and "easy to use on a live filesystem" administration tools are really lacking here.</p> <p>And just to highlight some of the usability issues that we sometimes face with Lustre.. In this particular instance:</p> <p> 1. The client <tt>mount</tt> command just hung without any helpful messages (maybe that's "correct" in this case).<br/> 2. The client console logs had this to say (after ctrl-c to kill the hung mount command). To an admin, what does this all mean???</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-04-21 14:17:54 LustreError: 17123:0:(lmv_obd.c:1292:lmv_statfs()) can't stat MDS #0 (lc2-MDT0000-mdc-ffff880d10805000), error -4 2014-04-21 14:17:54 LustreError: 16722:0:(lov_obd.c:937:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1 2014-04-21 14:17:54 LustreError: 16722:0:(lov_obd.c:937:lov_cleanup()) Skipped 26 previous similar messages 2014-04-21 14:17:54 Lustre: Unmounted lc2-client </pre> </div></div> <p> 3. lctl ping worked just fine from client to MDS, and from MDS to client<br/> 4. lctl list_nids on the MDS listed <b>only</b> the "correct" NID of the MDS<br/> 5. lctl list_nids on the client listed the client's NID correctly</p> <p>So, even to a seasoned Lustre admin, everything "seemed" to be configured correctly (and actually worked depending on the client's network being used). The fact that I had to rely on my past 3 years of Lustre experience and the lustre debug logs to track this down, is troubling at best. I'm not trying to bash Lustre here, I just want try and make my constant frustrations a little more known.</p> <p>Actually, in this case trying Cliff's suggestion while the mount command was hung (though with "mdc.*.import" instead of osc) might have shown the bad NID. </p> <p>I agree those error messages are unhelpful. (Not sure I want to know what happens when deathrow=1.)</p> <blockquote> <p>Can I make a feature request for a tool to read the configuration information from the MGS? Perhaps the llog_reader is a good place to start (as Niu suggested), but mounting the MGS devices as ldiskfs doesn't seem like a "correct" solution. Proper and "easy to use on a live filesystem" administration tools are really lacking here.</p></blockquote> <p>You can dump the client config file by debugfs on a live system. (debugfs -c -R "dump CONFIGS/lustre-client tmpfile" $mgs_device)</p> <blockquote><p>You can dump the client config file by debugfs on a live system. (debugfs -c -R "dump CONFIGS/lustre-client tmpfile" $mgs_device)</p></blockquote> <p>So the answer to Prakash's question is: "no, there is no sane way to do that".</p> <p>You basically need to be a Lustre developer to figure anything out about how an existing filesystem is configured, and that is just not acceptable this far into Lustre's development. It is going to make it very difficult for us to remain competitive with other filesystems if we don't start addressing the administration debt in Lustre.</p> <p>I changed the "summary" of this ticket to convert it from a question into a task that needs to be worked. Seeing how our sysadmins couldn't answer this question, and even Prakash struggled to get the information is a really great example of where work is needed for those of us that have been working on Lustre a long time.</p> <p>The command I mentioned should be run on the MGS, not on a client. And I would agree with the comments above, we have needed a better way to get config information for quite awhile. </p> <p>Ben Evans (bevans@cray.com) uploaded a new patch: <a href="http://review.whamcloud.com/23395" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23395</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> utils: Add scripts to output MGS config parameters<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d908640381ef779c6c9f2abad1adb168130a88ee</p> <p>For whatever reason, I don't recall ever seeing this ticket, nor the patch that was pushed against it.</p> <p>There was a command added many years ago "<tt>lctl --device %MGS llog_print \$&lt;llog_name&gt;</tt>" that will dump the current config llog (e.g. <tt>testfs-client</tt> or <tt>testfs-OST0001</tt>) from a running MGS.  In 2.6 this was changed to dump the llog in YAML format, for ease of reading/parsing (the pre-2.6 format was a huge mess and mostly unusable) via patch <a href="http://review.whamcloud.com/4254" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4254</a> "<a href="https://jira.whamcloud.com/browse/LU-2149" title="Make lctl llog commands to be used from scripts" class="issue-link" data-issue-key="LU-2149"><del>LU-2149</del></a> llog: lctl to work with llogs from command line".  In 2.10 the <tt>llog_print</tt> command was simplified to allow "<tt>lctl --device MGS llog_print &lt;llog_name&gt;</tt>" (i.e. without '%' before the device name, and without '$' before <tt>llog_name)</tt> via patch <a href="https://review.whamcloud.com/25468" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25468</a> "<a href="https://jira.whamcloud.com/browse/LU-9153" title="improve FID/config name specification for lctl llog_print and llog_info" class="issue-link" data-issue-key="LU-9153"><del>LU-9153</del></a> utils: improve llog name parsing".</p> <p>Ideally, the offline <tt>llog_reader</tt> tool would be improved to also dump the config llogs in the same YAML format (possibly with a <tt>--yaml</tt> flag to keep compatibility with the old format, if that is considered important).</p> <p>What is really needed at this point is to be able to take the YAML data as input and apply the tuning parameters to a system after a <tt>--writeconf</tt> or if the filesystem was reformatted.</p> <p>Ben Evans (bevans@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/31620" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31620</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> obdclass: llog_print params file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c2c410974cec3bcf1665b2c9bf9a85679e1e3a65</p> <p>Ben Evans (bevans@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/31846" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> utils: allow configuration through yaml files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 27084dd994954bc3a9c9a139c67a071bb786c0ce</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31620/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31620/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> obdclass: llog_print params file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 320e736191cce766cd0838ebbd5e76f1aefa1a6f</p> <p>Ben Evans (bevans@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/31931" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31931</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> tests: fix typo in sanity.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 005206fd806c7bbc525ca34151f0442ad4a3f971</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31931/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31931/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> tests: fix typo in sanity.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 47db24368d4191df91c80ad9ed60a62fa26922b5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/31846/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31846/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> utils: allow configuration through yaml files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8961f2d8e263c831ebebc47081d8e8ac61ffbcd9</p> <p>Landed for 2.12</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34250" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34250</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> obdclass: llog_print params file<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: a0a13a6981bee02af102c00b2cf08717fd2f6bab</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34250/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34250/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4939" title="Need to be able to sanely query and change MGS configuration information" class="issue-link" data-issue-key="LU-4939"><del>LU-4939</del></a> obdclass: llog_print params file<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: fb77f09ac8907059552eb5d606e9f8bcab580426</p> Related LU-11566 LU-11684 LU-12529 Development End date Thu, 26 Jun 2014 01:21:57 +0000 Rank 1|hzwkpr: Rank (Obsolete) 13651 Severity Start date Tue, 22 Apr 2014 01:21:57 +0000