https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9737 Lustre <p>Hi, </p> <p>I am test multi-rail in 2.10.0-RC1 and add one netwrok with ib0,ib1 interfaces.<br/> But lnetctl/lctl always hung after add one net.<br/> Should I enable anything ? Does anyone have user guide for multi-rail configuration? Thanks.</p> <p><span class="error">&#91;test steps&#93;</span><br/> 1.mdoprobe lnet<br/> 2.lnetctl lnet configure<br/> 3.lnetctl net show<br/> net:</p> <ul class="alternate" type="square"> <li>net: lo<br/> nid: 0@lo<br/> status: up</li> </ul> <p>4.lnetctl net add --net o2ib0 --if ib0,ib1<br/> 5.lnetctl net show =&gt;hung</p> <p><span class="error">&#91;kernel message&#93;</span><br/> [ 434.309534] LNet: Added LNI 172.20.110.220@o2ib <span class="error">&#91;8/256/0/180&#93;</span><br/> [ 434.323740] LNet: Added LNI 172.20.110.221@o2ib <span class="error">&#91;8/256/0/180&#93;</span><br/> [ 726.028183] INFO: task lctl:12235 blocked for more than 120 seconds.<br/> [ 726.028248] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> [ 726.028306] lctl D ffffffffa0cdc7b0 0 12235 10719 0x00000084<br/> [ 726.028313] ffff88086c75fd40 0000000000000086 ffff88086cb69f60 ffff88086c75ffd8<br/> [ 726.028317] ffff88086c75ffd8 ffff88086c75ffd8 ffff88086cb69f60 ffffffffa0cdc7a8<br/> [ 726.028320] ffffffffa0cdc7ac ffff88086cb69f60 00000000ffffffff ffffffffa0cdc7b0<br/> [ 726.028324] Call Trace:<br/> [ 726.028343] <span class="error">&#91;&lt;ffffffff8168c769&gt;&#93;</span> schedule_preempt_disabled+0x29/0x70<br/> [ 726.028348] <span class="error">&#91;&lt;ffffffff8168a3c5&gt;&#93;</span> __mutex_lock_slowpath+0xc5/0x1c0<br/> [ 726.028354] <span class="error">&#91;&lt;ffffffff8168982f&gt;&#93;</span> mutex_lock+0x1f/0x2f<br/> [ 726.028372] <span class="error">&#91;&lt;ffffffffa0c96fd6&gt;&#93;</span> LNetNIInit+0x46/0xa40 <span class="error">&#91;lnet&#93;</span><br/> [ 726.028388] <span class="error">&#91;&lt;ffffffffa0cb478f&gt;&#93;</span> lnet_ioctl+0x4f/0x250 <span class="error">&#91;lnet&#93;</span><br/> [ 726.028404] <span class="error">&#91;&lt;ffffffffa0c384ac&gt;&#93;</span> libcfs_ioctl+0x2ac/0x4c0 <span class="error">&#91;libcfs&#93;</span><br/> [ 726.028415] <span class="error">&#91;&lt;ffffffffa0c34517&gt;&#93;</span> libcfs_psdev_ioctl+0x67/0xf0 <span class="error">&#91;libcfs&#93;</span><br/> [ 726.028422] <span class="error">&#91;&lt;ffffffff81211ed5&gt;&#93;</span> do_vfs_ioctl+0x2d5/0x4b0<br/> [ 726.028427] <span class="error">&#91;&lt;ffffffff8121cb77&gt;&#93;</span> ? __fd_install+0x47/0x60<br/> [ 726.028431] <span class="error">&#91;&lt;ffffffff81212151&gt;&#93;</span> SyS_ioctl+0xa1/0xc0<br/> [ 726.028437] <span class="error">&#91;&lt;ffffffff816965c9&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> [ 846.028188] INFO: task lctl:12235 blocked for more than 120 seconds.<br/> [ 846.028256] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.</p> lustre: 2.10.0-RC1<br/> lnet: 0.7.0 LU-9737

lnetctl net show command hung after add net

Question/Request Minor Open Unresolved Amir Shehata sebg-crd-pm Wed, 5 Jul 2017 09:19:05 +0000 Fri, 14 Jul 2017 03:25:05 +0000 0 6 <p>The Multi-Rail instructions are in the manual - <a href="http://doc.lustre.org/lustre_manual.xhtml#lnetmr" class="external-link" target="_blank" rel="nofollow noopener">http://doc.lustre.org/lustre_manual.xhtml#lnetmr</a></p> <p>Amir</p> <p>Could you please assist with any follow on questions relating to the instructions in the manual?</p> <p>Thanks</p> <p>Peter</p> <p>I'm unable to reproduce your problem locally.</p> <p>Is this reproducible a 100% of the time? From the stack trace it appears that the ln_api_mutex is not being unlocked causing a deadlock. But I don't see a problem in the code.</p> <p>How did you get 2.10-RC1? did you build it yourself? or did you download the RPMs from somewhere?</p> <p>Is there other users trying to run "lctl" commands at the same time when you encounter this problem?</p> <p>Do you have lustre up? or are you loading lnet by itself?</p> <p>Can you also paste the output of the following command:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lnetctl -h </pre> </div></div> <p>Hi,</p> <p>#Is this reproducible a 100% of the time? <br/> =&gt;Yes.<br/> #How did you get 2.10-RC1? did you build it yourself? or did you download the RPMs from somewhere?<br/> =&gt;I got it from <a href="https://git.hpdd.intel.com/?p=fs/lustre-release.git" class="external-link" target="_blank" rel="nofollow noopener">https://git.hpdd.intel.com/?p=fs/lustre-release.git</a>, and build it by myself.<br/> #Is there other users trying to run "lctl" commands at the same time when you encounter this problem?<br/> =&gt;No, it only used by me.<br/> #Do you have lustre up? or are you loading lnet by itself?<br/> =&gt;I have also try to use "lctl net up" or "lnet start". They all got the same issue.<br/> I got the message when execute "lnet status", Is it allright?<br/> error: get_param: param_path 'health_check': No such file or directory<br/> running<br/> #Can you also paste the output of the following command:<br/> =&gt;<span class="error">&#91;root@mdsb1 ~&#93;</span># lnetctl -h<br/> Try interactive use without arguments or use one of:<br/> "lnet"<br/> "route"<br/> "net"<br/> "routing"<br/> "set"<br/> "import"<br/> "export"<br/> "stats"<br/> "peer_credits"<br/> "help"<br/> "exit"<br/> "quit"<br/> as argument.</p> <p>Becasue I have installed lustre 2.9 in these nodes before install lustre 2.10-RC1.<br/> In order to clarify the problem, I will try to install Lustre 2.10-RC1 in one pure OS node.<br/> Or you can provide me diagnostic script to get anything as you want to know. </p> <p> Another question, if "lnetctl net add --net o2ib0 --if ib0,ib1" works fine in mgs node, <br/> How can I format mds/oss with mgsnode NID or mount lustre with mgsnode NID ? Thanks.<br/> (Is it need included both mgs ib0,ib1 NIDs, like<br/> "mkfs.lustre --reformat --mdt --index=0 --mgsnode=172.20.110.220@o2ib:172.20.110.221@o2ib" ? <br/> mount.lustre 172.20.110.220@o2ib:172.20.110.221@o2ib:/hpcfs /mnt/client ? )</p> <p>Hi Amir Shehata,<br/> I found there is no lnetctl in my built lustre. The lnetctl file maybe installed by IEEL lustre before.<br/> I can setup lnetctl ok now after install yaml-devel package and rebuid lustre. Thanks. </p> <p>Another question, if "lnetctl net add --net o2ib0 --if ib0,ib1" works fine in mgs node, <br/> How can I format mds/oss with mgsnode NID or mount lustre with mgsnode NID ? Thanks.<br/> (Is it need included both mgs ib0,ib1 NIDs, like<br/> "mkfs.lustre --reformat --mdt --index=0 --mgsnode=172.20.110.220@o2ib:172.20.110.221@o2ib ..." ? <br/> mount.lustre 172.20.110.220@o2ib:172.20.110.221@o2ib:/hpcfs /mnt/client ? )</p> <p>Hi Amir, this looks like a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9729" title="missing mutex unlock in lnet_dyn_add_net()" class="issue-link" data-issue-key="LU-9729"><del>LU-9729</del></a></p> <p>Based on my analysis of the source code and the procedure that created the hang, it is almost certain that this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9729" title="missing mutex unlock in lnet_dyn_add_net()" class="issue-link" data-issue-key="LU-9729"><del>LU-9729</del></a>.</p> <p>Please note that even if the fix for <a href="https://jira.whamcloud.com/browse/LU-9729" title="missing mutex unlock in lnet_dyn_add_net()" class="issue-link" data-issue-key="LU-9729"><del>LU-9729</del></a> is included, the real problem is using an older <tt>lnetctl</tt>.</p> <p>To ensure that <tt>lnetctl</tt> will be built, install the rpms for <tt>libyaml</tt> and <tt>libyaml-devel</tt> on the build machine. This is not a hard requirement at the moment, but it will be in the future, because you need an up-to-date <tt>lnetctl</tt> to enable and configure new functionality like LNet Multi-Rail.</p> <p>Remember that to use <tt>lnetctl</tt> the <tt>libyaml</tt> rpm also has to be installed on all nodes.</p> <p>Thanks for your kind reminder. =&gt; Remember that to use lnetctl the libyaml rpm also has to be installed on all nodes.</p> Development Rank 1|hzzg3b: Rank (Obsolete) 9223372036854775807