https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7926 Lustre <p>We had our MDS become idle no with no errors on the console or logs. It responds extremely slow (takes mins to just do an ls). This has happened twice with 24 hours. We took crash dump in both cases. The crash dump show all mdt threads sitting in ' qsd_op_begin'<br/> like this</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 14754 TASK: ffff88400a83c040 CPU: 7 COMMAND: <span class="code-quote">"mdt01_002"</span> #0 [ffff883f634cd500] schedule at ffffffff81565692 #1 [ffff883f634cd5d8] schedule_timeout at ffffffff81566572 #2 [ffff883f634cd688] qsd_op_begin at ffffffffa0d04909 [lquota] #3 [ffff883f634cd738] osd_declare_qid at ffffffffa0d88449 [osd_ldiskfs] #4 [ffff883f634cd798] osd_declare_inode_qid at ffffffffa0d88702 [osd_ldiskfs] #5 [ffff883f634cd7f8] osd_declare_object_create at ffffffffa0d65d53 [osd_ldiskfs] #6 [ffff883f634cd858] lod_declare_object_create at ffffffffa0f4b482 [lod] #7 [ffff883f634cd8b8] mdd_declare_object_create_internal at ffffffffa0fa78cf [mdd] #8 [ffff883f634cd918] mdd_declare_create at ffffffffa0fbb4ce [mdd] #9 [ffff883f634cd988] mdd_create at ffffffffa0fbc631 [mdd] #10 [ffff883f634cda88] mdo_create at ffffffffa0e88058 [mdt] #11 [ffff883f634cda98] mdt_reint_open at ffffffffa0e923f4 [mdt] #12 [ffff883f634cdb78] mdt_reint_rec at ffffffffa0e7a481 [mdt] #13 [ffff883f634cdb98] mdt_reint_internal at ffffffffa0e5fed3 [mdt] #14 [ffff883f634cdbd8] mdt_intent_reint at ffffffffa0e6045e [mdt] #15 [ffff883f634cdc28] mdt_intent_policy at ffffffffa0e5dc3e [mdt] #16 [ffff883f634cdc68] ldlm_lock_enqueue at ffffffffa075e2c5 [ptlrpc] #17 [ffff883f634cdcd8] ldlm_handle_enqueue0 at ffffffffa0787ebb [ptlrpc] #18 [ffff883f634cdd48] mdt_enqueue at ffffffffa0e5e106 [mdt] #19 [ffff883f634cdd68] mdt_handle_common at ffffffffa0e62ada [mdt] #20 [ffff883f634cddb8] mds_regular_handle at ffffffffa0e9f505 [mdt] #21 [ffff883f634cddc8] ptlrpc_server_handle_request at ffffffffa07b70c5 [ptlrpc] #22 [ffff883f634cdea8] ptlrpc_main at ffffffffa07b989d [ptlrpc] #23 [ffff883f634cdf48] kernel_thread at ffffffff8100c28a </pre> </div></div> <p>I am attaching backtrace from the 2 crash dumps and also to lustre debug dumps. </p> LU-7926

MDS sits idle with extreme slow response to clients

Bug Blocker Resolved Duplicate Niu Yawei Mahmoud Hanafi Sun, 27 Mar 2016 18:32:26 +0000 Tue, 25 Oct 2016 14:10:59 +0000 Thu, 12 May 2016 12:55:26 +0000 Lustre 2.5.3 0 8 <p>Niu</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>Looks lots of MDT threads (creating objects) were trying to acquire quota for id 11361, but they all waited on an in-flight DQACQ. And I found lots of OSTs were acquiring quota lock from quota master, probably MDT was busy on these acquire request and the DQACQ from MDT wasn't processed in time?</p> <p>Are the OSTs just get rebooted (or network to MDT just recovered) when this happened? I'm wondering why there are so many requests from different OSTs to acquire quota locks.</p> <p>The OST/MDT and networks were not having any issues. UID 11361 user was running a 30K core job and they have an inode quota of 600K. They currently have over 530K files which is over their softquota of 500K.</p> <p>I have sent a request to the user to get additional info about their job.</p> <p>So it sounds like the user who was over their inode soft quota was creates a lot of files. Could this be a scalling issue in inode quota acquire? </p> <p>The user creates 90K files during their 30K core run.</p> <blockquote> <p>So it sounds like the user who was over their inode soft quota was creates a lot of files. Could this be a scalling issue in inode quota acquire?</p></blockquote> <p>Right, from the log we can see the user was creating lots of files. I don't think it's a scaling problem, because the root cause is a that former DQACQ from MDT didn't get replied somehow, but unfortunately that part of log is missed.</p> <p>My speculation is: There was a network hiccup caused all OSTs lost connections then reconnect, because there were tons of quota lock acquire from OSTs (pure quota lock acquire should only happen when quota slave start or reconnect to master). And probably that hiccup caused the DQACQ reply being lost?</p> <p>Going through the logs there is nothing to indicate that there was a lost of connections between OST, MDT or clients. There was a high load on a number of OSSes at 9:27:00. Are there any quota allocation adjusted once your over your inode softlimit?</p> <p>Ah, I think it's a livelock problem, from the stack trace we can see that all mdt service threads were busy on sending DQACQ requests and no available threads to handle the requests. This has been fixed in master by <a href="https://jira.whamcloud.com/browse/LU-6433" title="MDS deadlock in qouta " class="issue-link" data-issue-key="LU-6433"><del>LU-6433</del></a>.</p> <p>port to b2_5_fe: <a href="http://review.whamcloud.com/#/c/19250/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/19250/</a></p> <p>I'd think that MDT shouldn't get blocked in this case at all - just return to the client with -EINPROGRESS or something?</p> <p>Alex, the problem is that the thread sending DQACQ has to wait for the RPC timeout, once it get timeout, it'll reply -EINPROGRESS to client. So literally it's not a deadlock but a livelock problem, our solution is to use different set of threads for sending and handling the DQACQ requests.</p> <p>well, exactly. the fact that we send an RPC to get another quota unit can be the point at which we interrupt and return -EINPROGRESS to the client? I guess a potential problem is to avoid a livelock where a single unit is ping-pong'ed among 2+ servers because nobody consumes that right away..</p> <blockquote> <p>well, exactly. the fact that we send an RPC to get another quota unit can be the point at which we interrupt and return -EINPROGRESS to the client? </p></blockquote> <p>That sounds doable, it could eliminate all the sync acquirings.</p> <blockquote> <p>I guess a potential problem is to avoid a livelock where a single unit is ping-pong'ed among 2+ servers because nobody consumes that right away..</p></blockquote> <p>I don't know exactly what your are referring, could you illustrate?</p> <p>I cherry-picked the b2_5_fe port of <a href="http://review.whamcloud.com/#/c/19250/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/19250/</a> that Nui Yawei commented on 30/Mar/16 8:21 PM:<br/> <a href="https://jira.whamcloud.com/browse/LU-6433" title="MDS deadlock in qouta " class="issue-link" data-issue-key="LU-6433"><del>LU-6433</del></a> quota: handle QUOTA_DQACQ in READPAGE po<br/> rtal</p> <p>Not in production yet, but in our 2.5.3 code now.</p> <p>I cherry-picked the patch to b2_7_fe.</p> <p>So is it ok to close this ticket as a dupliate of <a href="https://jira.whamcloud.com/browse/LU-6433" title="MDS deadlock in qouta " class="issue-link" data-issue-key="LU-6433"><del>LU-6433</del></a>? </p> Related LU-6433 Development Rank 1|hzy5sn: Rank (Obsolete) 9223372036854775807 Severity