https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4572 Lustre <p>met threads hug. Forced reboot of mds 2 different time. </p> <p>uploading the following to ftp site:<br/> lustre-log.1391239242.7851.txt.gz <br/> vmcore-dmesg.txt.gz</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: MGS: haven't heard from client c546719d-1bcc-571f-a4e3-17f67dc35b50 (at 10.151.31.4@o2ib) in 199 seconds. I think it's dead, and I am evicting it. exp ffff880fd0587800, cur 1391236411 expire 1391236261 last 1391236212 LNet: Service thread pid 7851 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 7851, comm: mdt01_055 Call Trace: [&lt;ffffffff815404c2&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81080610&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa04156d1&gt;] cfs_waitq_timedwait+0x11/0x20 [libcfs] [&lt;ffffffffa06d201d&gt;] ldlm_completion_ast+0x4ed/0x960 [ptlrpc] [&lt;ffffffffa06cd790&gt;] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc] [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa06d1758&gt;] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] [&lt;ffffffffa06d1b30&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] [&lt;ffffffffa0dd7a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] [&lt;ffffffffa0dddc0c&gt;] mdt_object_lock0+0x28c/0xaf0 [mdt] [&lt;ffffffffa0dd7a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] [&lt;ffffffffa06d1b30&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] [&lt;ffffffffa0dde534&gt;] mdt_object_lock+0x14/0x20 [mdt] [&lt;ffffffffa0dde5a1&gt;] mdt_object_find_lock+0x61/0x170 [mdt] [&lt;ffffffffa0e0c80c&gt;] mdt_reint_open+0x8cc/0x20e0 [mdt] [&lt;ffffffffa043185e&gt;] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] [&lt;ffffffffa06fadcc&gt;] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] [&lt;ffffffffa05921b0&gt;] ? lu_ucred+0x20/0x30 [obdclass] [&lt;ffffffffa0dd7015&gt;] ? mdt_ucred+0x15/0x20 [mdt] [&lt;ffffffffa0df31cc&gt;] ? mdt_root_squash+0x2c/0x410 [mdt] [&lt;ffffffffa0df7981&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0ddcb03&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0ddd090&gt;] mdt_intent_reint+0x1f0/0x530 [mdt] [&lt;ffffffffa0ddaf3e&gt;] mdt_intent_policy+0x39e/0x720 [mdt] [&lt;ffffffffa06b2831&gt;] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] [&lt;ffffffffa06d91ef&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] [&lt;ffffffffa0ddb3c6&gt;] mdt_enqueue+0x46/0xe0 [mdt] [&lt;ffffffffa0de1ad7&gt;] mdt_handle_common+0x647/0x16d0 [mdt] [&lt;ffffffffa0e1b615&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa070b3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [&lt;ffffffffa04155de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa0426d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa0702729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [&lt;ffffffff81055813&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa070c75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] [&lt;ffffffffa070bc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa070bc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffffa070bc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1391239242.7851 </pre> </div></div> our lustre source tree is at:<br/> <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a> LU-4572

hung mdt threads

Bug Critical Resolved Duplicate Oleg Drokin Mahmoud Hanafi Sat, 1 Feb 2014 23:42:05 +0000 Wed, 6 Jan 2016 20:22:56 +0000 Wed, 6 Jan 2016 20:22:56 +0000 Lustre 2.4.1 0 9 <p>also uploaded bt of all tasks: bt.all.lu-4572.gz</p> <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>This looks like it may be the same issue as <a href="https://jira.whamcloud.com/browse/LU-2419" title="mdt threads stuck in ldlm_expired_completion_wait" class="issue-link" data-issue-key="LU-2419"><del>LU-2419</del></a> (the stack traces are very similar).</p> <p>Do you need any additional info to debug the issue?</p> <p>What's your at_max?</p> <p>Also: do you run 2.4 clients from that source as well? Can you please show me dmesg from e.g. 10.151.17.136 during all of this happening?<br/> Did you just crashdump the node to get all of this info or did you let it stay alive and just live-dumped?</p> <p>I have attached the console logs from 10.151.17.136 (r445i5n0.conlog.gz). </p> <p>We have both 2.4 and 2.1.5 clients.</p> <p>at_max = 600. </p> <p>The info was taken from crash dump. </p> <p>Well, the timeout you posted in the ticket seems to be related to client 10.151.31.4@o2ib being unresponsive, since the client is unresponsive, it does not send any lock cancels too I assume, so the locks it holds could not be cacelled and others who need said locks are forced to wait.<br/> I also see a number of network errors in logs (and client logs too), are you having some network problems there?</p> <p>Further on I see a different kind of problem:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&lt;4&gt;Lustre: 65757:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1391239042/real 1391239042] req@ffff8806c8f5d800 x1458619173294584/t0(0) o601-&gt;nbp7-MDT0000-lwp-MDT0000@0@lo:12/10 lens 336/336 e 3 to 1 dl 1391240399 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 &lt;4&gt;Lustre: 65757:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages &lt;4&gt;Lustre: 65757:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (1110:247s); client may timeout. req@ffff8806d18a2000 x1458067638265204/t0(0) o101-&gt;446c8dd2-5b53-9b80-51b7-1573c85a021b@10.151.17.177@o2ib:0/0 lens 616/544 e 3 to 0 dl 1391240152 ref 1 fl Complete:/0/0 rc 301/301 &lt;4&gt;LNet: Service thread pid 7918 completed after 1357.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). </pre> </div></div> <p>So here we are trying to send a quota rpc over local-connected lwp and it's timing out? that's not sounding right. Also the timeout for this local connection is huge for whatever reason. Also I do not see it being assigned any sane value too.</p> <p>We can see that overtime the timeout there grows even more.</p> <p>The other worrying sign is this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;Clocksource tsc unstable (delta = -68719428120 ns). Enable clocksource failo ver by adding clocksource_failover kernel parameter. &lt;4&gt;Lustre: 6577:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 76s req@ffff880f7ded2850 x1447159886197772/t0(0) o400-&gt;449796f8-1d26- c76e-7df6-0123fb984781@10.151.57.111@o2ib:0/0 lens 192/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 &lt;4&gt;Lustre: mdt: This server is not able to keep up with request traffic (cpu-bou nd). </pre> </div></div> <p>So 70+ seconds to handle request_in code is way too excessive. When this happens do you experience huge cpu utilization.<br/> Also I see the tsc unstable message - I think kernel uses that as internal timestamp counter, so if it deviates this hugely, could it be all internal timeouts are just not working properly? Is this something you can address just to be sure?</p> <p>One more question, how many OSTs do you have in your system?</p> <p>1. I am working through the logs to see if there were any IB fabric issues.</p> <p>2. When the system was in that state cpu utilization was minimal. </p> <p>3. We have seen the clocksource in our logs but we haven't found it to cause any issue. But I will switch it to hpet. </p> <p>4. This filesystem has 84 OSTs. </p> <p>I see. Let's see if switching to HPET helps any, I guess.<br/> With no cpu usage when this happened I do not see why things would be delayed by 70 seconds other than a broken clock.</p> <p>please close not seen the issue again.</p> <p>ok thanks Mahmoud</p> <p>I reopened this ticket because I tend to think ldlm_expired_completion_wait() should return an error code at least on client side so that the client enqueue process won't be stuck on completion AST forever.</p> <p>The same issue is seen on <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>.</p> <p>I decided to fix this issue in the ticket U-7372</p> Duplicate LU-2419 Related LU-2944 LU-7372 Development Rank 1|hzwe33: Rank (Obsolete) 12492 Severity