https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11125 Lustre <p>Servers were restarted and appeared to recover normally.  They briefly appeared to be handling the same (heavy) workload from before they were powered off, then started logging the "system was overloaded" message.  The kernel then reported several stacks like this:</p> <p>INFO: task ll_ost00_007:108440 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> ll_ost00_007 D ffff8ba4dc35bf40 0 108440 2 0x00000080<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffaad38919&gt;&#93;</span> schedule_preempt_disabled+0x39/0x90<br/> <span class="error">&#91;&lt;ffffffffaad3654f&gt;&#93;</span> __mutex_lock_slowpath+0x10f/0x250<br/> <span class="error">&#91;&lt;ffffffffaad357f2&gt;&#93;</span> mutex_lock+0x32/0x42<br/> <span class="error">&#91;&lt;ffffffffc1669afb&gt;&#93;</span> ofd_create_hdl+0xdcb/0x2090 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1322007&gt;&#93;</span> ? lustre_msg_add_version+0x27/0xa0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc132235f&gt;&#93;</span> ? lustre_pack_reply_v2+0x14f/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1322691&gt;&#93;</span> ? lustre_pack_reply+0x11/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc138653a&gt;&#93;</span> tgt_request_handle+0x92a/0x1370 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc132db5b&gt;&#93;</span> ptlrpc_server_handle_request+0x23b/0xaa0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc132b26b&gt;&#93;</span> ? ptlrpc_wait_event+0xab/0x350 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffaa6d5c32&gt;&#93;</span> ? default_wake_function+0x12/0x20<br/> <span class="error">&#91;&lt;ffffffffaa6cb01b&gt;&#93;</span> ? __wake_up_common+0x5b/0x90<br/> <span class="error">&#91;&lt;ffffffffc1331c70&gt;&#93;</span> ptlrpc_main+0xae0/0x1e90 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffc1331190&gt;&#93;</span> ? ptlrpc_register_service+0xe30/0xe30 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffaa6c0ad1&gt;&#93;</span> kthread+0xd1/0xe0<br/> <span class="error">&#91;&lt;ffffffffaa6c0a00&gt;&#93;</span> ? insert_kthread_work+0x40/0x40<br/> <span class="error">&#91;&lt;ffffffffaad44837&gt;&#93;</span> ret_from_fork_nospec_begin+0x21/0x21<br/> <span class="error">&#91;&lt;ffffffffaa6c0a00&gt;&#93;</span> ? insert_kthread_work+0x40/0x40</p> <p> And lustre began reporting:<br/> LustreError: 108448:0:(ofd_dev.c:1627:ofd_create_hdl()) lquake-OST0003:<span class="error">&#91;27917288460&#93;</span> destroys_in_progress already cleared</p> <p> </p> lustre-2.10.4_1.chaos-1.ch6.x86_64 servers <br/> RHEL 7.5 <br/> DNE1 file system <br/> LU-11125

ofd_create_hdl() destroys_in_progress already cleared

Bug Major Resolved Fixed Mikhail Pershin Olaf Faaland llnl Fri, 6 Jul 2018 07:08:17 +0000 Thu, 3 Jan 2019 19:17:17 +0000 Sat, 1 Sep 2018 05:57:16 +0000 Lustre 2.10.4 Lustre 2.12.0 Lustre 2.10.6 0 4 <p>Note this is not the same system as the one we reported an issue with earlier today.</p> <p>At the same time this was occurring, an aarch64 system running a 2.11 client was trying to connect unsuccessfully with these servers (the file system is mounted on the 2.11 nodes but the map_on_demand setting was changed on the servers and the connection attempts are all failing).</p> <p>/tmp/lustre-log.XXX files were being created, with contents like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000800:00000100:13.0:1530842270.070848:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00000800:00000100:13.0:1530842271.070330:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:13.0:1530842271.070337:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00000800:00000100:13.0:1530842272.070954:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:13.0:1530842272.070957:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00000800:00000100:13.0:1530842273.071332:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:13.0:1530842273.071336:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00000800:00000100:5.0:1530842274.071750:0:70765:0:(o2iblnd_cb.c:490:kiblnd_rx_complete()) Rx from 172.19.1.23@o2ib100 failed: 5 00000800:00000100:13.0:1530842274.071790:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:13.0:1530842274.071793:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00002000:00020000:9.0:1530842274.090444:0:102598:0:(ofd_dev.c:1627:ofd_create_hdl()) lquake-OST0002:[28991030285] destroys_in_progress already cleared 00000100:00000400:8.0:1530842274.090453:0:102596:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:203s); client may timeout. req@ffff8bb9b7e93850 x1605203337942624/t0(0) o5-&gt;lquake-MDT0003-mdtlov_UUID@172.19.1.114@o2ib100:296/0 lens 432/400 e 0 to 0 dl 1530842071 ref 1 fl Complete:/0/0 rc 0/0 00000800:00000100:13.0:1530842275.071508:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: reconnect (invalid service id), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 00000800:00000100:13.0:1530842275.071516:0:114359:0:(o2iblnd_cb.c:2672:kiblnd_rejected()) 172.19.1.23@o2ib100 rejected: no listener at 987 00000800:00000400:8.0:1530842276.068989:0:70764:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 172.19.1.23@o2ib100: 25 seconds 00000800:00000100:13.0:1530842276.070940:0:114359:0:(o2iblnd_cb.c:2646:kiblnd_check_reconnect()) 172.19.1.23@o2ib100: don't reconnect (no need), 12, 12, msg_size: 4096, queue_depth: 8/-1, max_frags: 256/-1 </pre> </div></div> <p>OST0002 and OST0003 both reported "destroys_in_progress already cleared", and both reported dumping /tmp/lustre-log files, but the node hosting OST0002 was rebooted and those log files were lost. So the content above is from a different OST, but it was also reporting the same stacks and console messages.</p> <p>Mike</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Olaf, are there any other bad effects except stack dumps and LustreError messages? Do OSTs works normal after all? You mentioned that OST0002 was rebooted, was that because of this issue or planned restart?</p> <p>Mike, at the time several IORs were running and they all appeared not to resume progress after recovery completed.  ltop was showing no activity on the file system, and the IORs were not finishing at the expected pace.</p> <p>Later, the problem seemed to resolve itself - all but one of the IORs completed successfully, but several took 10-20 times longer than they should have.  I'm looking into that now to get more information.</p> <p>One compute node crashed, but due to an unrelated problem we have neither the stack nor the crash dump.  We're looking into that problem as well.</p> <p> </p> <p> </p> <p> </p> <p>I was mistaken about OST0002 being rebooted a second time.  The file system was restarted just once, and the symptoms described here were seen after they came up and completed recovery.</p> <p> </p> <p>I am going to attempt to re-create this without the complication of the Lustre 2.11 arm system mounting the file system.</p> <p>That message - "destroys_in_progress already cleared" - related just to MDT-&gt;OST reconnects and may happen when MDT reconnects due to some reasons but OST is still performing orphan destroys from previous connect. I see in logs in description this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00002000:00020000:9.0:1530842274.090444:0:102598:0:(ofd_dev.c:1627:ofd_create_hdl()) lquake-OST0002:[28991030285] destroys_in_progress already cleared 00000100:00000400:8.0:1530842274.090453:0:102596:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:203s); client may timeout. req@ffff8bb9b7e93850 x1605203337942624/t0(0) o5-&gt;lquake-MDT0003-mdtlov_UUID@172.19.1.114@o2ib100:296/0 lens 432/400 e 0 to 0 dl 1530842071 ref 1 fl Complete:/0/0 rc 0/0 </pre> </div></div> <p>Looks like second thread <span class="error">&#91;102598&#93;</span> was waiting for the first <span class="error">&#91;102596&#93;</span> to complete OST_CREATE request with orphan destroys. And that first one took quite long time. It seems the problem is in that long orphan destroy, something was blocking it for quite a time. That can be just result of overall high load upon OST restart </p> <p>Thank you for the explanation.</p> <blockquote><p>It seems the problem is in that long orphan destroy, something was blocking it for quite a time. That can be just result of overall high load upon OST restart</p></blockquote> <p>It sounds like destroys_in_progress should be a CDEBUG message, since it can occur under normal operating conditions.</p> <p> </p> <p>It still may indicate the problem there, there are two different operations are possible on OST restart - MDT initiates orphan destroys, but also new precreates may be initiated by write replays. Both uses the same mutex 'os_create_lock' though one process destroys non-used objects and other creates new objects. That causes high contention on that mutex exactly on OST restart and I am checking now is that possible to separate locking for both processes. That would relieve load on that mutex.</p> <p>Hello Mikhail, have you learned anything more of interest?  Thanks.</p> <p>Olaf, this message is not an error so it can be just made less noisy as first step and then I will create new improvement ticket to investigate this area thoughtfully to lower contention on that mutex during OST recovery</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32985" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32985</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11125" title="ofd_create_hdl() destroys_in_progress already cleared" class="issue-link" data-issue-key="LU-11125"><del>LU-11125</del></a> ofd: decrease message level<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 68c740bcc319d019ec545eb50f25417a4bd9f7a9</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32985/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32985/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11125" title="ofd_create_hdl() destroys_in_progress already cleared" class="issue-link" data-issue-key="LU-11125"><del>LU-11125</del></a> ofd: decrease message level<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c25c4d60ac5fbede50a41da12f91de6e402b3c68</p> <p>This first patch landed for 2.12. Mike, can you please link in the ticket tracking the subsequent improvement?</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33100" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33100</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11125" title="ofd_create_hdl() destroys_in_progress already cleared" class="issue-link" data-issue-key="LU-11125"><del>LU-11125</del></a> ofd: decrease message level<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d266dbc4d62ab018d345159bb388f7f44fdca9d3</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33100/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33100/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11125" title="ofd_create_hdl() destroys_in_progress already cleared" class="issue-link" data-issue-key="LU-11125"><del>LU-11125</del></a> ofd: decrease message level<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 483e2e745c5294ae6dc2b6f85cc8976eb9f90fa3</p> <p>Is there a ticket for reducing the contention on os_create_lock? Thanks.</p> <blockquote><p>Is there a ticket for reducing the contention on os_create_lock? Thanks.</p></blockquote> <p><a href="https://jira.whamcloud.com/browse/LU-11399" title="use separate locks for orphan destroy and objects re-create at OFD" class="issue-link" data-issue-key="LU-11399">LU-11399</a></p> Related LU-11399 Development Rank 1|hzzytr: Rank (Obsolete) 9223372036854775807 Severity