https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8500 Lustre <p>Hit what looks like <a href="https://jira.whamcloud.com/browse/LU-4772" title="MGS is waiting for obd_unlinked_exports" class="issue-link" data-issue-key="LU-4772"><del>LU-4772</del></a> (MGS is waiting for obd_unlinked_exports) while running 2.7.2. So this is a duplicate case, but opening it for tracking to get the patch backported to the FE branch for NASA.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 15 09:19:12 nbp1-mds kernel: INFO: task umount:37486 blocked for more than 120 seconds. Aug 15 09:19:12 nbp1-mds kernel: Tainted: G -- ------------ T 2.6.32-573.26.1.el6.20160517.x86_64.lustre272 #1 Aug 15 09:19:13 nbp1-mds kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 15 09:19:13 nbp1-mds kernel: umount D 0000000000000003 0 37486 1 0x00000080 Aug 15 09:19:13 nbp1-mds kernel: ffff881f07cabab8 0000000000000086 0000000000000000 ffff881f07caba58 Aug 15 09:19:13 nbp1-mds kernel: ffff881f07caba18 ffff883e20b05400 001305aa0b19e806 0000000000000000 Aug 15 09:19:13 nbp1-mds kernel: ffff881f0b3ece4c 000000023ecf1493 ffff881e28d7d068 ffff881f07cabfd8 Aug 15 09:19:13 nbp1-mds kernel: Call Trace: Aug 15 09:19:13 nbp1-mds kernel: [&lt;ffffffff81574ce2&gt;] schedule_timeout+0x192/0x2e0 Aug 15 09:19:13 nbp1-mds kernel: [&lt;ffffffff810892c0&gt;] ? process_timeout+0x0/0x10 Aug 15 09:19:13 nbp1-mds kernel: [&lt;ffffffffa05b2296&gt;] obd_exports_barrier+0xb6/0x190 [obdclass] Aug 15 09:19:13 nbp1-mds kernel: [&lt;ffffffffa0e7ea14&gt;] mgs_device_fini+0x134/0x5b0 [mgs] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05d7122&gt;] class_cleanup+0x562/0xd20 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05b4216&gt;] ? class_name2dev+0x56/0xe0 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05d8e4a&gt;] class_process_config+0x156a/0x1ad0 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05d1205&gt;] ? lustre_cfg_new+0x435/0x630 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05d9525&gt;] class_manual_cleanup+0x175/0x4c0 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05b4216&gt;] ? class_name2dev+0x56/0xe0 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa061827f&gt;] server_put_super+0x9df/0x1060 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff811ad166&gt;] ? invalidate_inodes+0xf6/0x190 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff8119127b&gt;] generic_shutdown_super+0x5b/0xe0 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff81191366&gt;] kill_anon_super+0x16/0x60 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffffa05db0b6&gt;] lustre_kill_super+0x36/0x60 [obdclass] Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff81191b07&gt;] deactivate_super+0x57/0x80 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff811b1acf&gt;] mntput_no_expire+0xbf/0x110 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff811b261b&gt;] sys_umount+0x7b/0x3a0 Aug 15 09:19:14 nbp1-mds kernel: [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b Aug 15 09:20:03 nbp1-mds kernel: Lustre: MGS is waiting for obd_unlinked_exports more than 512 seconds. The obd refcount = 5. Is it stuck? </pre> </div></div> LU-8500

lustre-2.7.2 hits MGS is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 5. Is it stuck?

Bug Minor Resolved Fixed Hongchao Zhang Nathan Dauchy Mon, 15 Aug 2016 16:21:21 +0000 Mon, 27 May 2019 16:54:32 +0000 Thu, 8 Sep 2016 19:40:17 +0000 Lustre 2.7.0 Lustre 2.9.0 0 8 <p>Note that I saw a similar messages (one with stack trace, one without) other servers. The difference being that these are for the MDT, not the MGS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 15 09:22:09 nbp6-mds kernel: Lustre: nbp6-MDT0000 is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 9. Is it stuck? </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 15 10:20:15 nbp8-mds1 kernel: INFO: task umount:55640 blocked for more than 120 seconds. Aug 15 10:20:15 nbp8-mds1 kernel: Not tainted 2.6.32-504.30.3.el6.20151008.x86_64.lustre271 #1 Aug 15 10:20:16 nbp8-mds1 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 15 10:20:16 nbp8-mds1 kernel: umount D 0000000000000002 0 55640 55625 0x00000080 Aug 15 10:20:16 nbp8-mds1 kernel: ffff880935d2da78 0000000000000082 0000000000000000 ffff880935d2da18 Aug 15 10:20:16 nbp8-mds1 kernel: ffff880935d2d9d8 ffff883ddf90bc00 001a700794410389 0000000000000000 Aug 15 10:20:16 nbp8-mds1 kernel: ffff883f224ace8c 00000002bb51a25e ffff880e76565ad0 ffff880935d2dfd8 Aug 15 10:20:16 nbp8-mds1 kernel: Call Trace: Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffff81562c32&gt;] schedule_timeout+0x192/0x2e0 Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffff81086900&gt;] ? process_timeout+0x0/0x10 Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05a8296&gt;] obd_exports_barrier+0xb6/0x190 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa0f9f2c2&gt;] mdt_device_fini+0x642/0x1010 [mdt] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05ad406&gt;] ? class_disconnect_exports+0x116/0x2f0 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05cd122&gt;] class_cleanup+0x562/0xd20 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05aa216&gt;] ? class_name2dev+0x56/0xe0 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05cee4a&gt;] class_process_config+0x156a/0x1ad0 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05c7205&gt;] ? lustre_cfg_new+0x435/0x630 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05cf525&gt;] class_manual_cleanup+0x175/0x4c0 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa05aa216&gt;] ? class_name2dev+0x56/0xe0 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffffa060e597&gt;] server_put_super+0xcf7/0x1060 [obdclass] Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffff811a67a6&gt;] ? invalidate_inodes+0xf6/0x190 Aug 15 10:20:16 nbp8-mds1 kernel: [&lt;ffffffff8118ab8b&gt;] generic_shutdown_super+0x5b/0xe0 Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffff8118ac76&gt;] kill_anon_super+0x16/0x60 Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffffa05d10b6&gt;] lustre_kill_super+0x36/0x60 [obdclass] Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffff8118b417&gt;] deactivate_super+0x57/0x80 Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffff811ab11f&gt;] mntput_no_expire+0xbf/0x110 Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffff811abc6b&gt;] sys_umount+0x7b/0x3a0 Aug 15 10:20:17 nbp8-mds1 kernel: [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b Aug 15 10:21:46 nbp8-mds1 kernel: Lustre: nbp8-MDT0000: Not available for connect from 10.151.55.23@o2ib (stopping) Aug 15 10:21:46 nbp8-mds1 kernel: Lustre: Skipped 47513 previous similar messages Aug 15 10:22:06 nbp8-mds1 kernel: Lustre: nbp8-MDT0000 is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 7. Is it stuck? </pre> </div></div> <p>Please confirm whether the fix in <a href="https://jira.whamcloud.com/browse/LU-4772" title="MGS is waiting for obd_unlinked_exports" class="issue-link" data-issue-key="LU-4772"><del>LU-4772</del></a> addresses both the MGS and MDT.</p> <p>Nathan</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-4772" title="MGS is waiting for obd_unlinked_exports" class="issue-link" data-issue-key="LU-4772"><del>LU-4772</del></a> fix is already in the branch you are running with so it seems like this must be something new</p> <p>Hongchao</p> <p>What do you suggest here?</p> <p>Peter</p> <p>Hi Nathan,</p> <p>Is the debug log available during umounting MDT? it will dump the exports in this case, and will help to find the cause of the issue.<br/> Thanks!</p> <p>I'm afraid I don't have additional debugging info... was in a rush to get things shut down so we could complete server and Lustre version upgrades during a maintenance window, so I just power cycled the servers that hit this issue.</p> <p>Peter,</p> <p>According to Jay, the lustre-2.7.2-1.1nasS_mofed32v1 build (what was running on "nbp1-mds" above) does indeed have the <a href="https://jira.whamcloud.com/browse/LU-4772" title="MGS is waiting for obd_unlinked_exports" class="issue-link" data-issue-key="LU-4772"><del>LU-4772</del></a> fix.</p> <p>However, the lustre-2.7.1-5.1nasS_mofed31v5 build (what was on "nbp8-mds1" above) did NOT have the fix. I presume neither did the "lustre-2.5.3-6nasS_mofed31v5" version running on "nbp6-mds" above. Therefore, the issues for the latter two (regarding MDT) can be ignored for now as we assume the <a href="https://jira.whamcloud.com/browse/LU-4772" title="MGS is waiting for obd_unlinked_exports" class="issue-link" data-issue-key="LU-4772"><del>LU-4772</del></a> fix will prevent those in the future.</p> <p>Only the first stack trace on nbp1-mds is of interest then.</p> <p>Thanks!</p> <p>Hongchao</p> <p>What is your assessment of the first stack trace?</p> <p>Peter</p> <p>I'm still investigating the problem of the first stack trace, and don't find the cause of the problem yet.<br/> and I'm trying to create a debug patch to try to locate the problem, it's a little complicated and still need more time on it.</p> <p>by investigating the code lines, I find some cases leaking the export reference, which could be related to the issue.<br/> <a href="http://review.whamcloud.com/#/c/22021/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/22021/</a></p> <p>Hongchao</p> <p>I recommend fixing this on master first, then backporting to 2.7 FE for NASA to use</p> <p>Regards</p> <p>Peter</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22031" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22031</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8500" title="lustre-2.7.2 hits MGS is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 5. Is it stuck?" class="issue-link" data-issue-key="LU-8500"><del>LU-8500</del></a> ldlm: fix export reference problem<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a8fc617b0edbd195d22da21f30370fa4b33e74c1</p> <p>We just hit this on 2.8. Can't unmount our OSTs.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22031/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22031/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8500" title="lustre-2.7.2 hits MGS is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 5. Is it stuck?" class="issue-link" data-issue-key="LU-8500"><del>LU-8500</del></a> ldlm: fix export reference problem<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 108339f1543fb006f4ddd16830e7266df0b46723</p> <p>Oleg, thanks for landing.</p> <p>Hongchao, Peter, now we just need the finalized and checked and landed patch for the 2.7 FE branch.</p> <p>We need one for b2_8_fe as well.</p> <p>Here are the back-ported patches for:<br/> Lustre b2_7_fe branch: <a href="http://review.whamcloud.com/22289" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22289</a><br/> Lustre b2_8_fe branch: <a href="http://review.whamcloud.com/22302" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22302</a></p> <p>Master fix landed for 2.9; Landings to maintenance branches tracked separately</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31139" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31139</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8500" title="lustre-2.7.2 hits MGS is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 5. Is it stuck?" class="issue-link" data-issue-key="LU-8500"><del>LU-8500</del></a> ldlm: fix export reference<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 38a66a3cf18cb15fe98823019d925f9717ec38fa</p> Duplicate Related Development Rank 1|hzyknr: Rank (Obsolete) 9223372036854775807 Severity