https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4733 Lustre <p>Hi,</p> <p>We are seeing all mdt threads on the MDS stuck in "cfs_waitq_wait". At the same time, we have a lot of rpc request (15k/s).<br/> Looking closely at the console and the 'bt' from crash, we can see that those threads are coming from qos_statfs_update() where they block in l_wait_event and never wake up.</p> <p>What is strange is that cfs_time_beforeq_64(max_age, obd-&gt;obd_osfs_age) should be true.</p> <p>This issue was hit 4 times during February.</p> <p>Please find attached the dmesg and 'foreach bt' outputs.</p> <p>Sebastien.</p> LU-4733

All mdt thread stuck in cfs_waitq_wait

Bug Critical Resolved Cannot Reproduce Niu Yawei Sebastien Buisson Fri, 7 Mar 2014 14:20:29 +0000 Thu, 2 Oct 2014 14:43:29 +0000 Thu, 2 Oct 2014 14:43:29 +0000 Lustre 2.1.6 0 11 <p>Looks like a dup of <a href="https://jira.whamcloud.com/browse/LU-4335" title="MDS hangs due to mdt thread hung/inactive" class="issue-link" data-issue-key="LU-4335"><del>LU-4335</del></a> ...</p> <p>Hi Bruno,</p> <p>I was not so sure as problem described in <a href="https://jira.whamcloud.com/browse/LU-4335" title="MDS hangs due to mdt thread hung/inactive" class="issue-link" data-issue-key="LU-4335"><del>LU-4335</del></a> seems to be related to network problems, whereas there is no trace of such here.</p> <p>Niu</p> <p>Could you please confirm whether this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4335" title="MDS hangs due to mdt thread hung/inactive" class="issue-link" data-issue-key="LU-4335"><del>LU-4335</del></a> or not?</p> <p>Thanks</p> <p>Peter</p> <p>This issue should have been fixed on master, please see lov_update_set() in master, set-&gt;set_completes are atomic which is safe.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void lov_update_set(struct lov_request_set *set, struct lov_request *req, int rc) { req-&gt;rq_complete = 1; req-&gt;rq_rc = rc; cfs_atomic_inc(&amp;set-&gt;set_completes); if (rc == 0) cfs_atomic_inc(&amp;set-&gt;set_success); wake_up(&amp;set-&gt;set_waitq); } </pre> </div></div> <p>and please see 2.1.x, set-&gt;set_completes is just integer which is not safe (yes, increments may be lost under race), which means lov_set_finished() may never return true, and qos_statfs_done() will never be called if this happened</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void lov_update_set(struct lov_request_set *set, struct lov_request *req, int rc) { req-&gt;rq_complete = 1; req-&gt;rq_rc = rc; set-&gt;set_completes++; if (rc == 0) set-&gt;set_success++; cfs_waitq_signal(&amp;set-&gt;set_waitq); } </pre> </div></div> <p>Liang, 2.1 has only one ptlrpcd thread to process the statfs requests, so I think the ++ operation should be safe. Probably statfs can't finish on certain OST?</p> <p>Sebastien, were there any anbnormal messages on OSTs? and I see following messages on MDT:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[655444.910448] Uhhuh. NMI received <span class="code-keyword">for</span> unknown reason 3d on CPU 0. [655444.916528] Do you have a strange power saving mode enabled? [655444.922337] Kernel panic - not syncing: NMI: Not continuing [655444.928060] Pid: 0, comm: swapper Not tainted 2.6.32-220.23.1.bl6.Bull.28.10.x86_64 #1 [655444.936173] Call Trace: [655444.938768] &lt;NMI&gt; [&lt;ffffffff814851a0&gt;] ? panic+0x78/0x143 [655444.944518] [&lt;ffffffff81488ff0&gt;] ? do_nmi+0x240/0x2b0 [655444.949801] [&lt;ffffffff81488830&gt;] ? nmi+0x20/0x30 [655444.954655] [&lt;ffffffff812a9831&gt;] ? intel_idle+0xb1/0x170 [655444.960200] &lt;&lt;EOE&gt;&gt; [&lt;ffffffff813aa267&gt;] ? cpuidle_idle_call+0xa7/0x140 [655444.967165] [&lt;ffffffff81001e06&gt;] ? cpu_idle+0xb6/0x110 [655444.972536] [&lt;ffffffff8146ff2a&gt;] ? rest_init+0x7a/0x80 [655444.977909] [&lt;ffffffff81adff8c&gt;] ? start_kernel+0x481/0x48d [655444.983721] [&lt;ffffffff81adf31e&gt;] ? x86_64_start_reservations+0x125/0x129 [655444.990651] [&lt;ffffffff81adf41c&gt;] ? x86_64_start_kernel+0xfa/0x109 </pre> </div></div> <p>This looks like a hardware error, could it cause system hang under load?</p> <p>Niu, my 2 cents this comes from a forced crash-dump attempt or a HA fencing via NMI ...</p> <p>The lustre version running at customer site is based on lustre 2.1.6 + the following list of patches</p> <ul class="alternate" type="square"> <li><a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a> check for all attributes validity in revalidate</li> <li><a href="https://jira.whamcloud.com/browse/LU-3071" title="kernel BUG at fs/jbd2/transaction.c:293 (jbd2_journal_start)" class="issue-link" data-issue-key="LU-3071"><del>LU-3071</del></a> call cfs_cap_raise() before journal start</li> <li><a href="https://jira.whamcloud.com/browse/LU-3929" title="2.1.6-&gt;2.4.1 rolling upgrade: lustre-MDT0000: recovery is timed out, evict stale exports" class="issue-link" data-issue-key="LU-3929"><del>LU-3929</del></a> deny LWP connection for system upgrade</li> <li><a href="https://jira.whamcloud.com/browse/LU-3941" title="(lov_lock.c:671:lov_lock_enqueue()) ASSERTION( sublock-&gt;cll_conflict == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-3941"><del>LU-3941</del></a> only modify extent of sub-lock and preserve flags</li> <li><a href="https://jira.whamcloud.com/browse/LU-4298" title="&#39;lfs setstripe&#39; with bad values creates an empty file that has no striping information" class="issue-link" data-issue-key="LU-4298"><del>LU-4298</del></a> do not create file with no striping info</li> <li><a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> fix memory corruption</li> <li><a href="https://jira.whamcloud.com/browse/LU-3311" title="Interop 2.1.5&lt;-&gt;2.4 failure on test suite sanity test_27B: (llite_lib.c:1792:ll_update_inode()) ASSERTION( lu_fid_eq(&amp;lli-&gt;lli_fid, &amp;body-&gt;fid1) ) failed" class="issue-link" data-issue-key="LU-3311"><del>LU-3311</del></a> mdt: handle -ENOENT &amp;&amp; !MDS_OPEN_CREAT in reint open</li> <li><a href="https://jira.whamcloud.com/browse/LU-4363" title="(llite_lib.c:1683:ll_update_inode()) ASSERTION( lu_fid_eq(&amp;lli-&gt;lli_fid, &amp;body-&gt;fid1) ) failed" class="issue-link" data-issue-key="LU-4363"><del>LU-4363</del></a> open: backport MDS_OPEN_BY_FID flag</li> <li><a href="https://jira.whamcloud.com/browse/LU-2665" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-2665"><del>LU-2665</del></a> refine <a href="https://jira.whamcloud.com/browse/LU-2665" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-2665"><del>LU-2665</del></a> patch for POSIX compliance</li> <li><a href="https://jira.whamcloud.com/browse/LU-1306" title="LBUG at (dlm_lock.c:213:ldlm_lock_add_to_lru_nolock()) ASSERTION(lock-&gt;l_resource-&gt;lr_type != LDLM_FLOCK failed" class="issue-link" data-issue-key="LU-1306"><del>LU-1306</del></a> Protect l_flags with locking to prevent race on signal reception</li> <li><a href="https://jira.whamcloud.com/browse/LU-2943" title="LBUG mdt_reconstruct_open()) ASSERTION( (!(rc &lt; 0) || (lustre_msg_get_transno(req-&gt;rq_repmsg) == 0)) )" class="issue-link" data-issue-key="LU-2943"><del>LU-2943</del></a> LBUG in mdt_reconstruct_open() (patch set #3)</li> <li>ORNL-22 general ptlrpcd threads pool support</li> <li><a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a> implement a NUMA aware ptlrpcd binding policy</li> <li><a href="https://jira.whamcloud.com/browse/LU-1110" title="MDS Oops in osd_xattr_get() during file open by FID" class="issue-link" data-issue-key="LU-1110"><del>LU-1110</del></a> MDS Oops in osd_xattr_get() during file open by FID</li> <li><a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> to much unreclaimable slab space</li> <li><a href="https://jira.whamcloud.com/browse/LU-2624" title="Stop of ptlrpcd threads is long" class="issue-link" data-issue-key="LU-2624"><del>LU-2624</del></a> ptlrpc fix thread stop</li> <li><a href="https://jira.whamcloud.com/browse/LU-2683" title="Client deadlock in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-2683"><del>LU-2683</del></a> client deadlock in cl_lock_mutex_get</li> <li><a href="https://jira.whamcloud.com/browse/LU-2665" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-2665"><del>LU-2665</del></a> keep resend flocks</li> </ul> <p>so:</p> <ul class="alternate" type="square"> <li>multi ptlrpd is build in</li> <li>while I haven't identify which patch went with the lov_update_set safe routine, we do have it.</li> <li>the backtrace pointed by Niu is the result of an hardware NMI that was initiated to take the dump (so this is a managed hardware error, not a consequence nor an unexpected error).</li> </ul> <p>Alexandre, any error messages on OSTs?</p> <p>Niu,<br/> To complete the previous note from Alexandre, multi ptlrpcd and lov_update_set() safe routine are coming from ORNL-22 patch, which is a Bull backport in lustre 2.1.6 of branch b2_2 ORNL-22 patch (commit 71350744).</p> <p>I have attached the logs provided by support team, corresponding to the two last crashes which occured on 2 and 3 of march.</p> <p>here under is a copy of the analyze they have done:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; struct lov_qos.lq_statfs_in_progress 0xffff881eff05a6b0 lq_statfs_in_progress = 1 crash&gt; struct lov_statfs_data 0xffff881e3b9a2b80 struct lov_statfs_data { lsd_oi = { oi_policy = { ..... oi_flags = 5, &lt;--- OBD_STATFS_PTLRPCD | OBD_STATFS_NODELAY crash&gt; p *ptlrpcds $1 = { pd_size = 6352, pd_index = 0, pd_nthreads = 32, pd_thread_rcv = { pc_flags = 258, pc_lock = { raw_lock = { slock = 0 } } ..... crash&gt; struct ptlrpc_request_set 0xffff881fa6c702c0 struct ptlrpc_request_set { set_refcount = { counter = 1 }, set_new_count = { counter = 0 }, set_remaining = { counter = 0 } ...... </pre> </div></div> <p>It seems rpc have been send but not response is received.</p> <p>In cb_statfs_update conditions are not satisfied</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lov_update_set(lovreq-&gt;rq_rqset, lovreq, rc); if (rc) GOTO(out, rc); out: if (lovreq-&gt;rq_rqset-&gt;set_oi-&gt;oi_flags &amp; OBD_STATFS_PTLRPCD &amp;&amp; lov_set_finished(lovreq-&gt;rq_rqset, 0)) { lov_statfs_interpret(NULL, lovreq-&gt;rq_rqset, lovreq-&gt;rq_rqset-&gt;set_success != lovreq-&gt;rq_rqset-&gt;set_count); qos_statfs_done(lov); </pre> </div></div> <p>Hi,</p> <p>Please find attached the error logs on the OSS.<br/> We cannot see errors by the time the problem occurred (first occurrence is on March 2nd at 09:45am and second occurrence on March 3rd at 10:04pm).</p> <p>Sebastien.</p> <p>I see some OSTs reported slow read in the log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">bcluster210.log-20140309.gz:2014-03-02 13:09:40 Lustre: 24804:0:(filter_io.c:475:filter_preprw_read()) scratch3-OST002c: slow start_page_read 30s bcluster210.log-20140309.gz:2014-03-02 13:09:40 Lustre: 24804:0:(filter_io.c:475:filter_preprw_read()) Skipped 30 previous similar messages </pre> </div></div> <p>Maybe the OST is too busy to reply the statfs in time?</p> <p>Will the MDT get out of stuck state after the statfs RPC timedout and the burst of create requests from client level off?</p> <p>I do not have answers to your questions, but given their time of occurrence the 'slow' messages seem to be completely independent of the problem when all mdt threads are stuck in "cfs_waitq_wait", right?</p> <blockquote> <p>I do not have answers to your questions, but given their time of occurrence the 'slow' messages seem to be completely independent of the problem when all mdt threads are stuck in "cfs_waitq_wait", right?</p></blockquote> <p>I don't know for sure.</p> <p>I just want to figure out if it's a deadlock(or missing wakeup) problem (if the threads recovered when system load getting lower, then it's unlikely a deadlock/missing wakeup problem).</p> <p>BTW: Is there anyway to check the code running on servers? (to make sure the all the patches were applied properly)</p> <p>&gt; BTW: Is there anyway to check the code running on servers? (to make sure the all the patches were applied properly)</p> <p>Sure, but what do you suggest? I mean, how can we proceed to 'check the code'?</p> <p>Could you upload the code to the whamcloud ftp server (just like update logs)? Or show me where I can checkout the code? Thanks.</p> <p>Hi,</p> <p>On Whamcloud's FTP (ftp.whamcloud.com), I have created a directory "<a href="https://jira.whamcloud.com/browse/LU-4733" title="All mdt thread stuck in cfs_waitq_wait" class="issue-link" data-issue-key="LU-4733"><del>LU-4733</del></a>" containing a file lustre_216_bull_sources.tgz.</p> <p>Sebastien.</p> <p>Thank you, Sebastien. The code looks correct to me.</p> <p>It's possible that connection between MDT to some OST is broken, so the statfs can't be updated, and since there were many ongoing create requests, lq_statfs_in_progress could be changed to 1 immediately after a statfs done, so the waiters will go back to wait on qos_statfs_ready().</p> <p>When the problem raise again, could you try to collect lustre log on MDT with D_QOS &amp; D_RPCTRACE enabled? Thank you.</p> <p>Hi,<br/> two new occurences of this issue were detected at TGCC on april 7 and 9.</p> <p>Several things were made :</p> <ul class="alternate" type="square"> <li>list of clients disconnected was made to attempt to access the filesystem, with a df/ls/touch. All attempts have failed.</li> <li>umount of device on MDS failed, the process umount stuck on cfs_waitq_wait</li> </ul> <p>The attached file "lctl-dk.tgz" contains the following traces:</p> <ul class="alternate" type="square"> <li>mantis518_dk.log (collected with an echo -1 &gt; /proc/sys/lnet/debug)</li> <li>mantis518_rpc.log (collected with an echo +other +rpctrace &gt; /proc/sys/lnet/debug)</li> </ul> <p>Patrick</p> <blockquote> <p>Several things were made :<br/> list of clients disconnected was made to attempt to access the filesystem, with a df/ls/touch. All attempts have failed.<br/> umount of device on MDS failed, the process umount stuck on cfs_waitq_wait</p></blockquote> <p>Could you provide the stack trace for all processes on MDS when umount the device?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000100:00000400:2.0F:1396898402.999140:0:12693:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff880f02f9b800 x1463563674626235/t0(0) o400-&gt;4b5b918b-511a-1159-a7ee-db25715da035@JO.BOO.PA.BLW@o2ib2:0/0 lens 192/0 e 0 to 0 dl 1396898407 ref 2 fl New:H/0/ffffffff rc 0/-1 00000100:00000400:2.0:1396898402.999154:0:12693:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff880f02f9bc00 x1463563600312609/t0(0) o400-&gt;23a5a711-176e-4fea-3690-e6e8f81b7ba3@JO.BOO.IW.WW@o2ib2:0/0 lens 192/0 e 0 to 0 dl 1396898407 ref 2 fl New:H/0/ffffffff rc 0/-1 00000100:00000400:2.0:1396898402.999165:0:12693:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff88091d277000 x1463563647245622/t0(0) o400-&gt;e081ff72-cc00-6425-c96f-b093a974f1a4@JO.BOO.WB.AW@o2ib2:0/0 lens 192/0 e 0 to 0 dl 1396898407 ref 2 fl New:H/0/ffffffff rc 0/-1 00000100:00000400:2.0:1396898402.999176:0:12693:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply req@ffff88091d277400 x1463563646360851/t0(0) o400-&gt;822cdf71-bac5-e141-3e77-a03c858db938@JO.BOO.PA.BBA@o2ib2:0/0 lens 192/0 e 0 to 0 dl 1396898407 ref 2 fl New:H/0/ffffffff rc 0/-1 </pre> </div></div> <p>JO.BOO.PA.BLW@o2ib2 is one of the OST? Looks the connections between MDT and some OSTs are not stable. I tend to think this is same issue of <a href="https://jira.whamcloud.com/browse/LU-4335" title="MDS hangs due to mdt thread hung/inactive" class="issue-link" data-issue-key="LU-4335"><del>LU-4335</del></a> &amp; <a href="https://jira.whamcloud.com/browse/LU-4195" title="MDT Slow with ptlrpcd using 100% cpu." class="issue-link" data-issue-key="LU-4195"><del>LU-4195</del></a>.</p> <p>Hi,<br/> I have attached the stack trace of all processes (2014/04/07) provided by on site support team.<br/> They also mentionned that JO.BOO.PA.xxx addresses in lctl-dk_rpc file are client's (compute nodes) adresses, NOT OSS.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">bcluster231.log-20140309.gz:2014-03-03 23:18:53 LustreError: 25883:0:(o2iblnd_cb.c:2992:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds bcluster231.log-20140309.gz:2014-03-03 23:18:53 LustreError: 25883:0:(o2iblnd_cb.c:3055:kiblnd_check_conns()) Timed out RDMA with JO.BOO.AL.IL@o2ib2 (151): c: 8, oc: 0, rc: 8 bcluster231.log-20140309.gz:2014-03-03 23:18:53 Mar 3 23:18:53 bcluster231 kernel: LustreError: 25883:0:(o2iblnd_cb.c:2992:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds bcluster231.log-20140309.gz:2014-03-03 23:18:53 Mar 3 23:18:53 bcluster231 kernel: LustreError: 25883:0:(o2iblnd_cb.c:3055:kiblnd_check_conns()) Timed out RDMA with JO.BOO.AL.IL@o2ib2 (151): c: 8, oc: 0, rc: 8 bcluster231.log-20140309.gz:2014-03-03 23:22:50 Lustre: scratch3-OST01df: haven<span class="code-quote">'t heard from client scratch3-MDT0000-mdtlov_UUID (at JO.BOO.AL.IL@o2ib2) in 902 seconds. I think it'</span>s dead, and I am evicting it. exp ffff880dc9df1000, cur 1393885370 expire 1393884770 last 1393884468 </pre> </div></div> <p>The OSS log showed that something wrong with network when the problem happend on MDS, could you check your network as Amir suggested in <a href="https://jira.whamcloud.com/browse/LU-4195" title="MDT Slow with ptlrpcd using 100% cpu." class="issue-link" data-issue-key="LU-4195"><del>LU-4195</del></a>.</p> <p>Hi,</p> <p>The customer has upgraded to Lustre 2.4, and since then is not able to reproduce.<br/> So this ticket can be closed.</p> <p>Thanks,<br/> Sebastien.</p> <p>ok thanks Sebastien</p> Development Rank 1|hzwh33: Rank (Obsolete) 13008 Severity