https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9904 Lustre <p>New error seen during test on el6.9. Not seen before. Seen during lustre_rmmod every time.</p> <p>Suspect due to recent landings on master.</p> LU-9904

LNetError: 16885:0:(peer.c:1786:lnet_peer_push_event()) Push Put from unknown 0@<0:0> (source 0@<0:0>)

Bug Minor Resolved Fixed Amir Shehata Bob Glossman Tue, 22 Aug 2017 22:19:52 +0000 Sat, 30 Sep 2017 06:53:25 +0000 Sat, 30 Sep 2017 04:48:21 +0000 Lustre 2.11.0 0 7 <p>I'll investigate.</p> <p>This could occur when there is a shutdown. Basically, there is a new mechanism now where events are pushed to the peers whenever there is a local network interface change. During shutdown, everything is being removed. So that's detected as a change. and some pushes might make it out. When an event is received for that push message the peer is looked up but because of the shutdown it would've been removed, and that error is printed out. We can probably suppress this error when LNet is shutting down. There are no other negative consequences indicated by this message.</p> <p>Your explanation sounds quite plausible. I think it would be a very good idea to suppress such errors during shutdown. As it is it looks quite alarming and happens at least once on every shutdown, at least for me.</p> <p>sure. I'll push in a patch.</p> <p>found a set of steps to reproduce without shutdown:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer 2: modprobe lnet lnetctl lnet configure lnetctl net add --net tcp --<span class="code-keyword">if</span> eth0,eth1 peer1 modprobe lnet lnetctl lnet configure lnetctl net add --net tcp --<span class="code-keyword">if</span> eth0,eth1 lnetctl discover 192.168.122.30@tcp # discover peer2 # in /<span class="code-keyword">var</span>/log/messages # Aug 24 17:49:54 MRtest01 kernel: LNetError: 3447:0:(peer.c:1786:lnet_peer_push_event()) Push Put from unknown 0@&lt;0:0&gt; (source 0@&lt;0:0&gt;) </pre> </div></div> <p>I've started hitting this after a recent update as well. We definitely shouldn't be printing "Error" on the console for something that happens during normal operation. I'm also seeing a new error during mount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sh sanity.sh client: executing check_logdir /tmp/test_logs/1505452045 client: ../libcfs/libcfs/libcfs options: 'libcfs_panic_on_lbug=0' Logging to shared log directory: /tmp/test_logs/1505452045 client: executing yml_node IOC_LIBCFS_GET_NI error 22: Invalid argument </pre> </div></div> <p>The mount itself continues fine, but as with the other error, we shouldn't be printing errors to the console/terminal during normal operations, as that tends to confuse users.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29026" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29026</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9904" title="LNetError: 16885:0:(peer.c:1786:lnet_peer_push_event()) Push Put from unknown 0@&lt;0:0&gt; (source 0@&lt;0:0&gt;)" class="issue-link" data-issue-key="LU-9904"><del>LU-9904</del></a> lnet: reduce logging severity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0732d936aa07e8ee129829e738b82f10189cd7ce</p> <p>Andreas, the reason for <br/> IOC_LIBCFS_GET_NI error 22: Invalid argument<br/> is because an lctl list_nids is being called before the modules are loaded</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">testvm.centos7: executing check_logdir /tmp/test_logs/1505518717 Logging to shared log directory: /tmp/test_logs/1505518717 testvm.centos7: executing yml_node IOC_LIBCFS_GET_NI error 22: Invalid argument &lt;----- call to lctl list_nids Client: Lustre version: 2.10.52_98_g8e75219_dirty MDS: Lustre version: 2.10.52_98_g8e75219_dirty OSS: Lustre version: 2.10.52_98_g8e75219_dirty Stopping clients: testvm.centos7 /mnt/lustre (opts:) Stopping clients: testvm.centos7 /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre &lt;--- modules (including lnet) are loaded </pre> </div></div> <p>I would suggest that <tt>IOC_LIBCFS_GET_NI error 22: Invalid argument</tt> is not a useful error to print if <tt>lctl list_nids</tt> failed, so that should be fixed up somehow.</p> <p>Secondly, I don't see where <tt>lctl list_nids</tt> is being called, or I'd suggest to avoid calling it if that doesn't make sense. The only places I see it are in <tt>sk_nodemap_setup()</tt>, which is only called if <tt>$SK_S2S</tt> is set, and in <tt>host_nids_address()</tt> which is only <tt>sanity.sh::test_217</tt> and not during mount that I can see.</p> <p>yaml.sh:yml_node_info() gets called before the modules are loaded. in this function</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">for</span> nw in $(lctl list_nids | grep -v @lo | cut -f 2 -d <span class="code-quote">'@'</span> | uniq); <span class="code-keyword">do</span> </pre> </div></div> <p>This doesn't look like it's something new. I'm not sure exactly how the test framework works to make a change there.</p> <p>I would rather not remove the error when list_nids fails, because it's useful for debugging. prints out why it failed. I think it makes more sense not to call lctl list_nids from the test unless the modules are loaded first.</p> <p>I'd be happy if you fixed the test script to not call <tt>list_nids</tt> before the modules are loaded.</p> <p>Even so, the error message itself doesn't explain anything to the user who runs "<tt>lctl list_nids</tt>". I'm not suggesting the error should be removed, but rather that it should be improved to be actually useful to the user:</p> <ul class="alternate" type="square"> <li>include <tt>lctl list_nids:</tt> at the start, so the reader knows which command is failing</li> <li>expand the message for EINVAL to suggest that the lnet module is not loaded</li> </ul> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29026/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29026/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9904" title="LNetError: 16885:0:(peer.c:1786:lnet_peer_push_event()) Push Put from unknown 0@&lt;0:0&gt; (source 0@&lt;0:0&gt;)" class="issue-link" data-issue-key="LU-9904"><del>LU-9904</del></a> lnet: reduce logging severity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7fc8037d61b29fa2ac93ab5fb30fccd9b1c0066c</p> <p>Landed for 2.11.</p> Related LU-9480 Development Rank 1|hzziuf: Rank (Obsolete) 9223372036854775807 Severity