https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5816 Lustre <p>when recovery takes prolonged time due to waiting for all of the clients to reconnect, the recovery task triggers a the hangcheck timer.</p> <p>There is likely a way to tell the kernel we are fine or otherwise have some sort of a ping happen to avoid this message that unnecessary scares users.</p> LU-5816

Silence misleading kernel message"task tgt_recov:XXX blocked for more than 120 seconds"

Bug Major Resolved Fixed Hongchao Zhang Oleg Drokin llnl Tue, 28 Oct 2014 04:32:12 +0000 Tue, 4 Aug 2015 17:11:37 +0000 Tue, 3 Feb 2015 18:17:24 +0000 Lustre 2.5.4 Lustre 2.7.0 0 12 <p>Oleg, can you please paste the stack into the bug, so we can see where it is blocked?</p> <p>Bob,<br/> Could you have a look at this one and comment please?<br/> Thank you!</p> <p>not a lot of info to work with here yet. the "blocked for more than" message comes from check_hung_task() in the kernel when some task is in an uninterruptible sleep for too long. need some clue as to where lustre is sleeping. don't know if there is any kernel internal API to delay or suppress these warnings.</p> <p>Oh, I see Andreas already requested a stack trace on the hang. I agree that would be very useful info.</p> <p>Messages during recovery like</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 844.502040] INFO: task tgt_recov:19764 blocked <span class="code-keyword">for</span> more than 120 seconds. [ 844.509756] Not tainted 2.6.32-431.17.1.el6.wc.x86_64 #1 [ 844.516404] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [ 844.525393] tgt_recov D 0000000000000003 0 19764 2 0x00000000 [ 844.533247] ffff881fcaed1da0 0000000000000046 0000000000000000 ffff881fcaed1d64 [ 844.541824] 0000009100000000 ffff88207fc28800 ffff88011c456880 0000000000000400 [ 844.550425] ffff881fcaecb058 ffff881fcaed1fd8 000000000000fbc8 ffff881fcaecb058 [ 844.559077] Call Trace: [ 844.561987] [&lt;ffffffffa07ca620&gt;] ? check_for_clients+0x0/0x70 [ptlrpc] [ 844.569538] [&lt;ffffffffa07cbc8d&gt;] target_recovery_overseer+0x9d/0x230 [ptlrpc] [ 844.577874] [&lt;ffffffffa07ca310&gt;] ? exp_connect_healthy+0x0/0x20 [ptlrpc] [ 844.587551] [&lt;ffffffff8109af00&gt;] ? autoremove_wake_function+0x0/0x40 [ 844.594898] [&lt;ffffffffa07d2550&gt;] ? target_recovery_thread+0x0/0x1920 [ptlrpc] [ 844.603226] [&lt;ffffffffa07d2a90&gt;] target_recovery_thread+0x540/0x1920 [ptlrpc] [ 844.611534] [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 [ 844.618690] [&lt;ffffffffa07d2550&gt;] ? target_recovery_thread+0x0/0x1920 [ptlrpc] [ 844.627004] [&lt;ffffffff8109ab56&gt;] kthread+0x96/0xa0 [ 844.632593] [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [ 844.638274] [&lt;ffffffff8109aac0&gt;] ? kthread+0x0/0xa0 [ 844.643947] [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>and</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 1805.700512] INFO: task tgt_recov:19764 blocked <span class="code-keyword">for</span> more than 120 seconds. [ 1805.708227] Not tainted 2.6.32-431.17.1.el6.wc.x86_64 #1 [ 1805.714870] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [ 1805.723897] tgt_recov D 000000000000000b 0 19764 2 0x00000000 [ 1805.731776] ffff881fcaed1da0 0000000000000046 0000000000000000 ffff883f9ac0ba21 [ 1805.740357] ffffc900802298f8 ffff88402a8af430 0000000000004ed2 0000000000004ece [ 1805.748926] ffff881fcaecb058 ffff881fcaed1fd8 000000000000fbc8 ffff881fcaecb058 [ 1805.757503] Call Trace: [ 1805.760426] [&lt;ffffffffa07cea60&gt;] ? check_for_next_transno+0x0/0x590 [ptlrpc] [ 1805.768553] [&lt;ffffffffa07cbc8d&gt;] target_recovery_overseer+0x9d/0x230 [ptlrpc] [ 1805.776888] [&lt;ffffffffa07ca330&gt;] ? exp_req_replay_healthy+0x0/0x30 [ptlrpc] [ 1805.784921] [&lt;ffffffff8109af00&gt;] ? autoremove_wake_function+0x0/0x40 [ 1805.792266] [&lt;ffffffffa07d2cba&gt;] target_recovery_thread+0x76a/0x1920 [ptlrpc] [ 1805.800565] [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 [ 1805.807732] [&lt;ffffffffa07d2550&gt;] ? target_recovery_thread+0x0/0x1920 [ptlrpc] [ 1805.816039] [&lt;ffffffff8109ab56&gt;] kthread+0x96/0xa0 [ 1805.821621] [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [ 1805.827287] [&lt;ffffffff8109aac0&gt;] ? kthread+0x0/0xa0 [ 1805.832961] [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>target_recovery_overseer() calls wait_event(), but it may take more than 120 seconds for the event to happen. The solution might be to use wait_event_timeout() in a loop to wake back up after 90% of sysctl_hung_task_timeout_secs has elapsed.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#/c/12672/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12672/</a></p> <p>While testing patch <a href="http://review.whamcloud.com/13046" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13046</a> on Lustre b2_5 branch, conf-sanity test 47 hung.</p> <p>On OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:13:45:INFO: task tgt_recov:3507 blocked for more than 120 seconds. 06:13:45: Not tainted 2.6.32-431.29.2.el6_lustre.gbb46ddc.x86_64 #1 06:13:45:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 06:13:45:tgt_recov D 0000000000000000 0 3507 2 0x00000080 06:13:45: ffff880066fdbda0 0000000000000046 0000000000000000 ffff880074f078c4 06:13:45: 00000053ffffff0a ffffffffa07f79a6 0000000000000064 0008000000010000 06:13:45: ffff880071556638 ffff880066fdbfd8 000000000000fbc8 ffff880071556638 06:13:45:Call Trace: 06:13:45: [&lt;ffffffff8109b2ce&gt;] ? prepare_to_wait+0x4e/0x80 06:13:45: [&lt;ffffffffa073b910&gt;] ? check_for_next_lock+0x0/0x270 [ptlrpc] 06:13:45: [&lt;ffffffffa073cc9d&gt;] target_recovery_overseer+0x9d/0x230 [ptlrpc] 06:13:45: [&lt;ffffffffa073b370&gt;] ? exp_lock_replay_healthy+0x0/0x30 [ptlrpc] 06:13:45: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 06:13:45: [&lt;ffffffffa074121e&gt;] target_recovery_thread+0x96e/0x1960 [ptlrpc] 06:13:45: [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 06:13:45: [&lt;ffffffffa07408b0&gt;] ? target_recovery_thread+0x0/0x1960 [ptlrpc] 06:13:45: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 06:13:45: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 06:13:45: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 06:13:45: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/97a07574-8473-11e4-8915-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/97a07574-8473-11e4-8915-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12672/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12672/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5816" title="Silence misleading kernel message&quot;task tgt_recov:XXX blocked for more than 120 seconds&quot;" class="issue-link" data-issue-key="LU-5816"><del>LU-5816</del></a> target: don't trigger watchdog waiting in recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ba0a1b36870807e8182189bcb08f7b105aff6c57</p> <p>Landed for 2.7</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13623" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13623</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5816" title="Silence misleading kernel message&quot;task tgt_recov:XXX blocked for more than 120 seconds&quot;" class="issue-link" data-issue-key="LU-5816"><del>LU-5816</del></a> target: don't trigger watchdog waiting in recovery<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: c1d6ebbaaf7d598622fd7a6319c97bb29ce68805</p> Related LU-5805 Development Rank 1|hzwzmn: Rank (Obsolete) 16310 Severity