https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5695 Lustre <p>Sometimes lc_watchdogd disappears w/o any messages and lustre logs are not dumped after watchdog triggered.</p> <p>How the correct behaviour should look:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: Service thread pid 7096 was inactive for 10.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 7096, comm: lctl Call Trace: [&lt;ffffffff81528eb2&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81084220&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa0380df7&gt;] proc_trigger_watchdog+0x67/0x80 [libcfs] [&lt;ffffffff811fd8e7&gt;] proc_sys_call_handler+0x97/0xd0 [&lt;ffffffff811fd934&gt;] proc_sys_write+0x14/0x20 [&lt;ffffffff81188f68&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81189861&gt;] sys_write+0x51/0x90 [&lt;ffffffff8152b2be&gt;] ? do_device_not_available+0xe/0x10 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b LustreError: dumping log to /tmp/lustre-log.1411548646.7096 </pre> </div></div> <p>and how it may look in the kernel logs when lustre logs are not dumped:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 242: Check that watchdog causes kernel log dump == 09:19:38 (1411550378) LNet: Service thread pid 12742 stopped after 20.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: DEBUG MARKER: sanity test_242: @@@@@@ FAIL: Lustre log wasn't dumped Lustre: DEBUG MARKER: == sanity test complete, duration 29 sec == 09:20:01 (1411550401) </pre> </div></div> LU-5695

watchdog dispatch thread disappears

Bug Major Resolved Fixed James A Simmons Alexander Zarochentsev patch Wed, 1 Oct 2014 13:26:47 +0000 Tue, 27 Feb 2018 04:28:31 +0000 Tue, 27 Feb 2018 04:28:31 +0000 Lustre 2.11.0 0 6 <p>After a closer look at libcfs/libcfs/watchdog.c it was found that <b>LCW_FLAG_STOP</b> flag in <b>lcw_flags</b> variable is only set and never gets cleared.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 17:33:12 ] $ git grep lcw_flags libcfs/libcfs/watchdog.c:static unsigned long lcw_flags = 0; libcfs/libcfs/watchdog.c: if (test_bit(LCW_FLAG_STOP, &amp;lcw_flags)) libcfs/libcfs/watchdog.c: if (test_bit(LCW_FLAG_STOP, &amp;lcw_flags)) { libcfs/libcfs/watchdog.c: set_bit(LCW_FLAG_STOP, &amp;lcw_flags); </pre> </div></div> <p> So if <b>lcw_refcount</b> reaches zero and the watchdog thread is stopped by lcw_dispatch_stop() , it will be never working again (it exists immediately after start) until the modules reload or system restart.</p> <p>the fix is like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/libcfs/libcfs/watchdog.c b/libcfs/libcfs/watchdog.c index ed1acf7..e71b48a 100644 --- a/libcfs/libcfs/watchdog.c +++ b/libcfs/libcfs/watchdog.c @@ -330,6 +330,7 @@ <span class="code-keyword">static</span> void lcw_dispatch_stop(void) wake_up(&amp;lcw_event_waitq); wait_for_completion(&amp;lcw_stop_completion); + clear_bit(LCW_FLAG_STOP, &amp;lcw_flags); CDEBUG(D_INFO, <span class="code-quote">"watchdog dispatcher has shut down.\n"</span>); </pre> </div></div> <p>a proper patch will be submitted soon</p> <p>patch <a href="http://review.whamcloud.com/#/c/12155/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12155/</a></p> <p>I will monitor this issue</p> <p>The patch has failed review, can you address the issues?</p> <p>Bug out of date, no patch update. Closing</p> <p>Please reopen. I plan to update the patch but I was waiting until the port to sysfs happens for Lustre 2.10.</p> <p>Still waiting for a patch</p> <p>procfs (or sysfs) part of the patch is only for testing, I think at least the actual fix from the patch can be landed.</p> <p>Old issue, already fixed</p> <p>Is this fixed?</p> <p>Since Alex is okay with a one line fix I refreshed the patch. Very simple and should be landed soon.</p> <p>yes, lets go with the one-line fix. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/12155/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/12155/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5695" title="watchdog dispatch thread disappears" class="issue-link" data-issue-key="LU-5695"><del>LU-5695</del></a> libcfs: watchdog dispatch thread fix<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1947bc08c0709ad80611dc65785ccb8dbf7f7214</p> <p>Landed for 2.11</p> Related LU-8066 Development Rank 1|hzwxi7: Rank (Obsolete) 15940 Severity