https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5428 Lustre <p>This problem was reported against a newly installed system at NOAA (Boulder). The system was idle at the time:</p> <p>Jul 17 04:53:57 lfs-mds-0-1 kernel: : LNet: Service thread pid 29363 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Jul 17 05:26:52 lfs-mds-0-1 kernel: : LNet: Service thread pid 29363 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Jul 20 04:10:08 lfs-mds-0-1 kernel: : LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. quota=off. Opts: <br/> Jul 21 01:20:12 lfs-mds-0-1 kernel: : LNet: Service thread pid 13603 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Jul 21 14:35:12 lfs-mds-0-1 kernel: : LNet: Service thread pid 13829 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Jul 23 05:55:12 lfs-mds-0-1 kernel: : LNet: Service thread pid 29363 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Jul 24 11:19:47 lfs-mds-0-1 kernel: : LNet: Service thread pid 13672 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).</p> <p>Customer stats that he is observing LNet: Service thread pid completed after 0.00s even when the system is idle (they are on testbed (pre-production)). </p> <p>I also saw these same messages on another idle system that was newly installed (Harvard (HMU)). </p> New Installation - Lustre 2.4.3 servers, 1.8.9 Clients LU-5428

LNet: Service thread pid completed after 0.00s (DDN SR34734)

Bug Minor Resolved Fixed Liang Zhen Oz Rentas Tue, 29 Jul 2014 20:10:37 +0000 Tue, 21 Mar 2017 13:54:52 +0000 Thu, 28 Aug 2014 19:52:39 +0000 Lustre 2.7.0 0 6 <p>Liang</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>It is strange that we saw "Service thread pid completed after 0.00s", because watchdog should complain only if service thread took too long to finish a request, but here we saw 0.00s. I think it could be a bug in our watchdog code, I will look into it. <br/> Btw, I guess the system should be still working fine besides these fault warning?</p> <p>Yes, very strange. I agree.</p> <p>&gt;&gt;Btw, I guess the system should be still working fine besides these fault warning?<br/> Yes</p> <p>Thanks,<br/> Oz</p> <p>Any ideas on this one?</p> <p>Hi, sorry for late response. I have worked out a patch: <a href="http://review.whamcloud.com/11415" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11415</a><br/> Briefly, reason of this issue is because there is a race between lc_watchdog_touch and lcw_cb which will generate false alarm, this should be harmless, but it is still good to fix it.</p> <p>Patch landed to master</p> <p>Thanks much. Go ahead and close this.</p> <p>Thanks Oz</p> <p>Will this patch work with earlier versions of Lustre? For example, version 2.1.5 ?</p> <p>In version 2.1.5, we are seeing the same symptoms:</p> <p>Feb 21 07:29:59 mmp-2 kernel: Lustre: Service thread pid 5040 was inactive for 0.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Feb 21 07:29:59 mmp-2 kernel: Pid: 5040, comm: ll_mgs_00<br/> Feb 21 07:29:59 mmp-2 kernel:<br/> Feb 21 07:29:59 mmp-2 kernel: Call Trace:<br/> Feb 21 07:29:59 mmp-2 kernel: Lustre: Service thread pid 5040 completed after 0.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa0620af4&gt;&#93;</span> ? target_send_reply_msg+0x54/0x190 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa03b360e&gt;&#93;</span> cfs_waitq_wait+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa065e9f9&gt;&#93;</span> ptlrpc_wait_event+0x2b9/0x2c0 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffff8105fa40&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa066657d&gt;&#93;</span> ptlrpc_main+0x61d/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa0665f60&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa0665f60&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffffa0665f60&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 21 07:29:59 mmp-2 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Feb 21 07:29:59 mmp-2 kernel:</p> <p>yes, I think it should work for 2.1.5.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23162" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23162</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5428" title="LNet: Service thread pid completed after 0.00s (DDN SR34734)" class="issue-link" data-issue-key="LU-5428"><del>LU-5428</del></a> libcfs: race in lc_watchdog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 637b75e8ebd2e17c151f32aecb341dbfa336264b</p> <p>Patch will be tracked for landing under <a href="https://jira.whamcloud.com/browse/LU-9235" title="LNet: Service thread pid 13033 was inactive for 0.00s." class="issue-link" data-issue-key="LU-9235"><del>LU-9235</del></a></p> Related Development Rank 1|hzwskn: Rank (Obsolete) 15111 Severity