https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6132 Lustre <p>Unloading IB drivers results in hung task message and driver unloading is stuck forever.</p> <p>Steps to reproduce:<br/> 1) Have a lustre mount to server<br/> 2) On server do /etc/init.d/openibd stop<br/> 3) openibd script is stuck<br/> 4) After 120 seconds, following message is seen in dmesg:</p> <p>LNetError: 131-3: Received notification of device removal<br/> Please shutdown LNET to allow this to proceed<br/> INFO: task modprobe:2837 blocked for more than 120 seconds.<br/> Not tainted 2.6.32_431.el6_lustre.x86_64 #1<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> modprobe D 0000000000000000 0 2837 2777 0x00000000<br/> ffff88011c649bf8 0000000000000082 00000000ffffffff 00000000ffffffff<br/> ffff88011c649c38 ffffffff81060b13 ffff88011c649c78 00000000811a591f<br/> ffff8800cc23f058 ffff88011c649fd8 000000000000fbc8 ffff8800cc23f058<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff81060b13&gt;&#93;</span> ? perf_event_task_sched_out+0x33/0x70<br/> <span class="error">&#91;&lt;ffffffff8105a570&gt;&#93;</span> ? __dequeue_entity+0x30/0x50<br/> <span class="error">&#91;&lt;ffffffff81528c25&gt;&#93;</span> schedule_timeout+0x215/0x2e0<br/> <span class="error">&#91;&lt;ffffffff81527d80&gt;&#93;</span> ? thread_return+0x4e/0x76e<br/> <span class="error">&#91;&lt;ffffffff815288a3&gt;&#93;</span> wait_for_common+0x123/0x180<br/> <span class="error">&#91;&lt;ffffffff81065df0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffff810686da&gt;&#93;</span> ? __cond_resched+0x2a/0x40<br/> <span class="error">&#91;&lt;ffffffff815289bd&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> <span class="error">&#91;&lt;ffffffffa03170be&gt;&#93;</span> cma_remove_one+0x18e/0x210 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa021f5ff&gt;&#93;</span> ib_unregister_device+0x4f/0x100 <span class="error">&#91;ib_core&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0257aa6&gt;&#93;</span> mlx4_ib_remove+0xc6/0x300 <span class="error">&#91;mlx4_ib&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0167881&gt;&#93;</span> mlx4_remove_device+0x71/0x90 <span class="error">&#91;mlx4_core&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa01679b3&gt;&#93;</span> mlx4_unregister_interface+0x43/0x80 <span class="error">&#91;mlx4_core&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa026f891&gt;&#93;</span> __exit_compat+0x15/0x69 <span class="error">&#91;mlx4_ib&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810b9454&gt;&#93;</span> sys_delete_module+0x194/0x260<br/> <span class="error">&#91;&lt;ffffffff8152d8ce&gt;&#93;</span> ? do_page_fault+0x3e/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>The cause of this is that ko2iblnd does not handle device removal (should probably handle it the same as disconnected event).</p> RHEL 6.5 with MLNX_OFED 2.3 and ConnectX3/ConnectX3 Pro/ConnectIB HW (but I&#39;m guessing is reproducible with any OS and any OFED/upstream kernel). LU-6132

Unable to unload ib drivers with lustre loaded

Bug Critical Open Unresolved Dmitry Eremin Yan Burman llnl Sun, 18 Jan 2015 15:11:11 +0000 Tue, 28 May 2019 15:46:04 +0000 Lustre 2.7.0 0 22 <p>Release 2.7 will have Dynamic LNet Config (DLC) which allows user to dynamically add/remove LNet NI in userspace, after that you should be able to unload ib drivers. So far we don’t plan to handle this event and trigger NI removal in kernel space.</p> <p>Could you please explain why you would want to be able to unload the IB module while Lustre is up and running?</p> <p>One scenario would be reboot. Also, device removal event will be received upon PCI reset to device or unbind/hotplug.<br/> From my experience on other projects, there are two changes that need to be done to support this:<br/> 1) Be able to differentiate between CM IDs of the listener and of connections<br/> 2) If CM ID is of a connection, do the same handling as disconnected event. In case of listener, just destroy the CM ID</p> <p>It seems to me that the correct solution here is during shutown to clean up the LNET routes/modules before unconfiguring IB. That should happen via the /etc/init.d/lnet script. Not clear why that isn't happening?</p> <p>The problem would happen if the mlx4/mlx5 drivers are unloaded before LNET is cleaned up or if device is removed (which is easily simulated by unloading mlx</p> {4,5} <p>_* modules).<br/> Fixing the script (assuming it's a script problem) will fix one scenario out of few. Handling the device removal event will be cleaner and handle other scenarios as well.<br/> Handling CM ID of a connection should be similar if not identical to handling of disconnected event.<br/> The only non-trivial (as far as I understood) part is identifying that the CM ID belongs to a listener, as it is not being saved currently from what I saw.<br/> What do you think of this idea?</p> <p>I think the shutdown scripts should be fixed to honor the correct dependency - i.e. shutdown the IB users (e.g. LNet) before any attempts to shutdown any part of IB. As to LNet support of device removal, if there's a valid use case for that we should certainly support it. But I'd tend to say incorrect shutdown order isn't a valid use case. If there's other scenarios where LNet would need to handle device removal, please point it out.</p> <p>Other use cases where you may get device removal event are:<br/> 1) Card/FW failure and reset issued on the card<br/> 2) VPI - changing in runtime between ethernet and IB port type<br/> 3) Unloading driver perhaps for maintenance<br/> 4) Hotplug of card (as well as VF in SRIOV case)</p> <p>2-4 could be handled by removing the IB NI first with DLC, since they are all admin actions. Case 1 could probably be cleaned up with DLC NI shutdown after it has happened.</p> <p>This issue has been reported by one of our customer. It usually occurs when shutting down a Lustre client while Lustre file systems are still mounted.</p> <p>We've encountered this problem as well on both lustre clients and lustre servers. Problem usually occurs, as Gregoire mentioned, when a lustre fs is mounted on a client or a lustre server has a target mounted during shut down.</p> <p>This happens when you have Lustre share mounted but try to unload OFED drivers. The shutdown sequence should be following:</p> <ol> <li>umount /mnt/lustre</li> <li>lustre_rmmod</li> <li>unload OFED drivers</li> </ol> <p>We can make first 2 steps in our shutdown script but cannot guarantee that OFED drivers will not be unloaded first.</p> <p> </p> <p>It is possible to make lustre aware when the IB core is unloaded but i haven't had cycles to implement this. I guess in that case we would have to force evictions of clients if that happens.</p> <p>I don't see why the client would need to be evicted, per-se if the IB interface is stopped. In theory, if the client has some other form of communication with the server (e.g. TCP or OPA) it could continue to work after the IB interface is stopped. Handling that cleanly would definitely need some development work, and is best left until after the LNet Multi-Rail code is landed, since I suspect it will need to deal with that situation in any case.</p> <p>One simple option for handling this in the short term is adding an <tt>/sbin/umount.lustre</tt> script which tries <tt>lustre_rmmod</tt> to unload the modules, but fails silently if the modules are in use (i.e. another filesystem is mounted). That would drop the LNet references and disconnect the client, allowing the IB modules to be unloaded. However, this depends on the client unmount happening before the IB modules are cleaned up. The other option is a systemd script (see patch <a href="http://review.whamcloud.com/21457" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21457</a> "<a href="https://jira.whamcloud.com/browse/LU-8384" title="convert startup scripts to systemd unit files for EL7" class="issue-link" data-issue-key="LU-8384"><del>LU-8384</del></a> scripts: Add scripts to systemd for EL7").</p> Duplicate LU-7755 Related LU-9439 LU-8384 LU-11066 Development Rank 1|hzx46v: Rank (Obsolete) 17090 Severity