https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11278 Lustre <p>After the LNet health merger I'm seeing a new bug on Power8 platforms. Currently I can't  even ping the MGS from my Power8 client. I see the following back trace:</p> <p><span class="error">&#91;  172.614537&#93;</span> <span class="error">&#91;c000001fffdc7180&#93;</span> <span class="error">&#91;c000000000c683a0&#93;</span> _raw_spin_unlock_bh+0x50/0x80</p> <p><span class="error">&#91;  172.614610&#93;</span> <span class="error">&#91;c000001fffdc71b0&#93;</span> <span class="error">&#91;c000000000a4d0e8&#93;</span> peernet2id+0x78/0xd0</p> <p><span class="error">&#91;  172.614670&#93;</span> <span class="error">&#91;c000001fffdc71f0&#93;</span> <span class="error">&#91;c000000000acd06c&#93;</span> netlink_broadcast_filtered+0x31c/0x740</p> <p><span class="error">&#91;  172.614749&#93;</span> <span class="error">&#91;c000001fffdc72b0&#93;</span> <span class="error">&#91;d00000000cc33298&#93;</span> rdma_nl_multicast+0x58/0x90 <span class="error">&#91;ib_core&#93;</span></p> <p><span class="error">&#91;  172.614826&#93;</span> <span class="error">&#91;c000001fffdc72f0&#93;</span> <span class="error">&#91;d00000000cc3a270&#93;</span> send_mad+0x4e0/0x6a0 <span class="error">&#91;ib_core&#93;</span></p> <p><span class="error">&#91;  172.614903&#93;</span> <span class="error">&#91;c000001fffdc7390&#93;</span> <span class="error">&#91;d00000000cc3bdcc&#93;</span> ib_sa_path_rec_get+0x21c/0x5b0 <span class="error">&#91;ib_core&#93;</span></p> <p><span class="error">&#91;  172.614977&#93;</span> <span class="error">&#91;c000001fffdc7460&#93;</span> <span class="error">&#91;d00000000fc629e4&#93;</span> path_rec_start+0xb4/0x190 <span class="error">&#91;ib_ipoib&#93;</span></p> <p><span class="error">&#91;  172.615051&#93;</span> <span class="error">&#91;c000001fffdc7500&#93;</span> <span class="error">&#91;d00000000fc65e1c&#93;</span> ipoib_start_xmit+0x63c/0x7e0 <span class="error">&#91;ib_ipoib&#93;</span></p> <p><span class="error">&#91;  172.615122&#93;</span> <span class="error">&#91;c000001fffdc75b0&#93;</span> <span class="error">&#91;c000000000a6906c&#93;</span> dev_hard_start_xmit+0xec/0x2f0</p> <p><span class="error">&#91;  172.615193&#93;</span> <span class="error">&#91;c000001fffdc7640&#93;</span> <span class="error">&#91;c000000000ab7ef4&#93;</span> sch_direct_xmit+0x164/0x260</p> <p><span class="error">&#91;  172.615264&#93;</span> <span class="error">&#91;c000001fffdc76e0&#93;</span> <span class="error">&#91;c000000000a69908&#93;</span> __dev_queue_xmit+0x698/0x9e0</p> <p><span class="error">&#91;  172.615335&#93;</span> <span class="error">&#91;c000001fffdc7790&#93;</span> <span class="error">&#91;c000000000a7900c&#93;</span> neigh_connected_output+0xfc/0x170</p> <p><span class="error">&#91;  172.615406&#93;</span> <span class="error">&#91;c000001fffdc77e0&#93;</span> <span class="error">&#91;c000000000a80194&#93;</span> neigh_update+0x644/0x790</p> <p><span class="error">&#91;  172.615465&#93;</span> <span class="error">&#91;c000001fffdc7860&#93;</span> <span class="error">&#91;c000000000b400a8&#93;</span> arp_process+0x2c8/0x850</p> <p><span class="error">&#91;  172.615525&#93;</span> <span class="error">&#91;c000001fffdc7940&#93;</span> <span class="error">&#91;c000000000b407cc&#93;</span> arp_rcv+0x19c/0x230</p> <p><span class="error">&#91;  172.615584&#93;</span> <span class="error">&#91;c000001fffdc79b0&#93;</span> <span class="error">&#91;c000000000a57c4c&#93;</span> __netif_receive_skb_core+0x73c/0x1010</p> <p><span class="error">&#91;  172.615694&#93;</span> <span class="error">&#91;c000001fffdc7a70&#93;</span> <span class="error">&#91;c000000000a5f8a8&#93;</span> netif_receive_skb_internal+0x58/0x160</p> <p><span class="error">&#91;  172.615831&#93;</span> <span class="error">&#91;c000001fffdc7ab0&#93;</span> <span class="error">&#91;c000000000a62e38&#93;</span> napi_gro_receive+0x1c8/0x2f0</p> <p><span class="error">&#91;  172.615983&#93;</span> <span class="error">&#91;c000001fffdc7af0&#93;</span> <span class="error">&#91;d00000000c8e85fc&#93;</span> mlx5i_handle_rx_cqe+0x20c/0x3c0 <span class="error">&#91;mlx5_core&#93;</span></p> <p><span class="error">&#91;  172.616158&#93;</span> <span class="error">&#91;c000001fffdc7ba0&#93;</span> <span class="error">&#91;d00000000c8e7878&#93;</span> mlx5e_poll_rx_cq+0x278/0xb50 <span class="error">&#91;mlx5_core&#93;</span></p> <p><span class="error">&#91;  172.616308&#93;</span> <span class="error">&#91;c000001fffdc7c30&#93;</span> <span class="error">&#91;d00000000c8e8a30&#93;</span> mlx5e_napi_poll+0x160/0xe50 <span class="error">&#91;mlx5_core&#93;</span></p> <p><span class="error">&#91;  172.616444&#93;</span> <span class="error">&#91;c000001fffdc7cf0&#93;</span> <span class="error">&#91;c000000000a62aec&#93;</span> net_rx_action+0x3bc/0x540</p> <p><span class="error">&#91;  172.616559&#93;</span> <span class="error">&#91;c000001fffdc7e00&#93;</span> <span class="error">&#91;c000000000c690cc&#93;</span> __do_softirq+0x14c/0x3dc</p> <p><span class="error">&#91;  172.616675&#93;</span> <span class="error">&#91;c000001fffdc7ef0&#93;</span> <span class="error">&#91;c0000000001423d4&#93;</span> irq_exit+0x1e4/0x1f0</p> <p><span class="error">&#91;  172.616791&#93;</span> <span class="error">&#91;c000001fffdc7f20&#93;</span> <span class="error">&#91;c000000000017190&#93;</span> __do_irq+0xa0/0x200</p> <p><span class="error">&#91;  172.616905&#93;</span> <span class="error">&#91;c000001fffdc7f90&#93;</span> <span class="error">&#91;c00000000002ea40&#93;</span> call_do_irq+0x14/0x24</p> <p><span class="error">&#91;  172.617019&#93;</span> <span class="error">&#91;c000000ffba43a40&#93;</span> <span class="error">&#91;c000000000017390&#93;</span> do_IRQ+0xa0/0x120</p> <p><span class="error">&#91;  172.617135&#93;</span> <span class="error">&#91;c000000ffba43aa0&#93;</span> <span class="error">&#91;c000000000008bd4&#93;</span> hardware_interrupt_common+0x114/0x120</p> Power8 running RHEL7 alt kernel. LU-11278

LNet failures on Power8

Bug Critical Closed Not a Bug Amir Shehata James A Simmons Thu, 23 Aug 2018 17:35:56 +0000 Wed, 19 Dec 2018 21:11:20 +0000 Wed, 19 Dec 2018 21:11:20 +0000 Lustre 2.12.0 0 4 <p>First problem is the YAML file I have is a empty peer section so I get:</p> <p>lnetctl import &lt; ~/lnet-multirail.config</p> <p>add:</p> <p>    - net:</p> <p>          errno: 0</p> <p>          descr: "success"</p> <p>    - peer:</p> <p>          errno: -2</p> <p>          descr: error copying nids from YAML block</p> <p>    - max_interfaces:</p> <p>          errno: 0</p> <p>          descr: "success" </p> <p>.......</p> <p> </p> <p>You can reproduce it with attached file.</p> <p> </p> <p>The empty peer issue is resolved with this ticket -  <a href="https://jira.whamcloud.com/browse/LU-11006" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11006</a></p> <p>The config attached doesn't have an empty peer section. I'm able to import it correctly. Also if there is an empty peer section, then the error listed above makes sense and it doesn't stop the rest of the elements from being configured. So that's not an issue.</p> <p>As regards to the stack trace listed, that's most likely an issue with the mlx5 driver having a problem when it's coming up. It's not related to the Health feature. I believe it's not reproducible.</p> <p>Regarding the YAML file that's not new behavior. And I think it's the correct behavior. It's simply being informative and telling you that something in the config file is not formed correctly or missing, which is the case.</p> <p>Unless there is a real issue here, I'd rather close this ticket as not a bug.</p> <p>**UPDATE:</p> <p>The fix Sonia points out is for not printing the "peer:" if there are no peers. As to regards the error while importing a file, I think is entirely valid.</p> <p>Just about the "peer section" issue -  if there are no peers configured on the node and we export the configuration to a yaml file, then the yaml file does have a "peer" root but because no peers are configured so no other information and just the root "peer" is there. <br/> And while importing that yaml file, it shows up the error which is misleading. So that issue does need to eb taken care of. But yes this is not LNet health related.</p> <p>I'm rebuilding new images with the 3 health patches and Sonia's peer config fix.</p> <p>If there are other issues please open a new ticket.</p> <p>I'm still seeing the mlx5 failures with the lastest LNet stack. I gave it another try and it failed the same way. Some how LNet is exposing a mlx5 bug</p> <p>Going to try patch <a href="https://patchwork.ozlabs.org/patch/764801" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.ozlabs.org/patch/764801</a></p> <p>I would suggest building a version without Health and try and reproduce the mlx5 issue. There hasn't been any changes in o2iblnd stack, except very minor flag setting that are only triggered when there is failure in sending traffic.</p> <p>Also, when does the mlx5 failure occur? Does it occur on bootup? Can you determine if it is hit before or after LNet is loaded? If you simply don't startup LNet, do you still see the issue? If it occurs only when LNet is loaded, can you determine a specific operation LNet is taking that triggers this scenario? I'm not clear about the order of operations from the info you've shared.</p> <p>I don't mind joining you in a debug session to try and figure this out.</p> <p>This might not be a Power8 only issue. The problem shows up when attempting to mount ZFS with a lustre version using the MOFED 4.4 stack. Can you try that setup to see if you can reproduce this problem.</p> Related LU-6387 Development Rank 1|i0017b: Rank (Obsolete) 9223372036854775807 Severity