https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9525 Lustre <p>The code for both "lctl network down" and "lnetctl net unconfigure" was written assuming the network was originally brought up using either "lctl network configure" or "lnetctl net configure".  If the network came up as part of a mount or "modprobe lustre", there is a flag which does not get set thereby preventing lctl or lnetctl from bringing down the network.</p> <p>This enforces a restriction that the mechanism which configures the network is the same one which unconfigures the network.  I see no reason for that restriction.  It is confusing and should not be there.</p> <p>This ticket has been opened to change that part of the code to let us switch to a different mechanism to unconfigure from the one used to configure.  Otherwise, debugging in the field is just no fun!</p> LU-9525

"lctl network down" won't work if network brought up by a mount

Bug Major Resolved Not a Bug Patrick Farrell Doug Oucharek Wed, 17 May 2017 21:42:57 +0000 Tue, 16 Apr 2019 14:45:58 +0000 Fri, 12 Apr 2019 14:25:35 +0000 0 7 <p>With Lustre 2.12.0 it doesn't appear there's any way for <tt>lctl net down</tt> to work:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># modprobe lnet # lctl list_nids IOC_LIBCFS_GET_NI error 100: Network is down # lctl net up # lctl list_nids 192.168.56.20@tcp # lctl net down LNET busy </pre> </div></div> <p>Debug log with ALL the debugging:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param debug=-1 debug=-1 # lctl dk &gt; /dev/null ; lctl net down; lctl dk LNET busy 00000400:00000001:1.0F:1550767099.121831:0:5433:0:(module.c:69:libcfs_ioctl()) Process entered 00000400:00000001:1.0:1550767099.121833:0:5433:0:(linux-module.c:113:libcfs_ioctl_getdata()) Process entered 00000400:00000010:1.0:1550767099.121834:0:5433:0:(linux-module.c:136:libcfs_ioctl_getdata()) alloc '(*hdr_pp)': 128 at ffffa0cd5829d900 (tot 502104). 00000400:00000001:1.0:1550767099.121835:0:5433:0:(linux-module.c:143:libcfs_ioctl_getdata()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:1.0:1550767099.121836:0:5433:0:(linux-module.c:91:libcfs_ioctl_data_adjust()) Process entered 00000400:00000001:1.0:1550767099.121837:0:5433:0:(linux-module.c:105:libcfs_ioctl_data_adjust()) Process leaving (rc=0 : 0 : 0) 00000400:00000080:1.0:1550767099.121837:0:5433:0:(module.c:90:libcfs_ioctl()) libcfs ioctl cmd 3221775672 00000400:00000010:1.0:1550767099.121839:0:5433:0:(module.c:118:libcfs_ioctl()) kfreed 'hdr': 128 at ffffa0cd5829d900 (tot 501976). 00000400:00000001:1.0:1550767099.121840:0:5433:0:(module.c:119:libcfs_ioctl()) Process leaving (rc=18446744073709551600 : -16 : fffffffffffffff0) Debug log: 9 lines, 9 kept, 0 dropped, 0 bad. </pre> </div></div> <p>Nathaniel,</p> <p>What happens if you do lustre_rmmod in there, like we discussed in Skype?  (Not saying you should have to, just curious what happens)</p> <p>Doing <tt>lustre_rmmod</tt> tears everything down.  Doing a partial teardown results in a panic (probably because of polling by iml-agent) see <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a></p> <p>So I spent a bit on this before I realized, as far as I can tell, the code is fine, and I can't reproduce this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># modprobe lnet # lctl list_nids IOC_LIBCFS_GET_NI error 100: Network is down # lctl net up # lctl list_nids 192.168.56.20@tcp # lctl net down LNET busy </pre> </div></div> <p>lctl net down works fine for me in this scenario, both on master and on b2_12.</p> <p>I also can't reproduce the problem Doug described about the config thing.  It's true that you cannot unconfigure LNet while ptlrpc is loaded, but that's because ptlrpc won't load without network interfaces available.  ptlrpc doesn't support being loaded without nis configured, so you can't unconfigure them without unloading it.</p> <p>But it's not just "a flag that's set" - that flag reflects whether LNet initialized its own config or if it was done by ptlrpc.  If it was done by ptlrpc, then LNet can't just tear it down.  All of the scenarios and orders I could think of for "config done by lnet, done by ptlrpc (this is the same as "done by a mount")" work fine, given that you accept that you can't unconfigure lnet while ptlrpc is loaded.</p> <p>Maybe there's just something IML is doing that's making LNet busy...?  Because this code seems to be fine.</p> <p>Sounds like this is expected behavior. I'll try using <tt>lustre_rmmod ptlrpc</tt> unconditionally before bringing down LNet.</p> <p>Per our conversation this morning, it looks like with <a href="https://jira.whamcloud.com/browse/LU-11986" title="After partial lustre_rmmod, lnet panics on debugfs read" class="issue-link" data-issue-key="LU-11986"><del>LU-11986</del></a> fixed (which is in progress, patch exists) IML is able to do what it needs to do.</p> Development Rank 1|hzzd1b: Rank (Obsolete) 9223372036854775807 Severity