https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14378 Lustre <p>Console log on node with MDT0005 (NID 172.19.3.1@o2ib600) reports:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 18990:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1609295691, 300s ago), entering recovery for MGS@172.19.3.1@o2ib600 ns: MGC172.19.3.1@o2ib600 lock: ffff97619d5133c0/0xa8790faf05ee75fc lrc: 4/1,0 mode: --/CR res: [0x68736c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x3481cb7270a3b1bc expref: -99 pid: 18990 timeout: 0 lvb_type: 0 LustreError: 18990:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message LustreError: 25121:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC172.19.3.1@o2ib600: namespace resource [0x68736c:0x2:0x0].0x0 (ffff972b9abea0c0) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 25121:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message </pre> </div></div> MDT <br/> 3.10.0-1160.4.1.1chaos.ch6.x86_64 <br/> lustre-2.12.5_10.llnl-3.ch6.x86_64 LU-14378

ldlm_resource_complain()) MGC172.19.3.1@o2ib600: namespace resource [0x68736c:0x2:0x0].0x0 (ffff972b9abea0c0) refcount nonzero (1) after lock cleanup; forcing cleanup.

Bug Minor Closed Fixed Mikhail Pershin Olaf Faaland llnll Thu, 28 Jan 2021 05:55:57 +0000 Wed, 11 Jan 2023 18:44:48 +0000 Sun, 16 Jan 2022 08:57:50 +0000 0 5 <p>For my records my local ticket is TOSS5036</p> <p>The day of the issue we see repeated communication failures with lsh-MDT000c, within the hours of 2am, 7am, and 9am:</p> <p>For example</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2020-12-29 02:40:20 [4122150.157428] LNetError: 12879:0:(o2iblnd_cb.c:3359:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 2020-12-29 02:40:20 [4122150.168998] LNetError: 12879:0:(o2iblnd_cb.c:3434:kiblnd_check_conns()) Timed out RDMA with 172.19.3.13@o2ib600 (11): c: 6, oc: 0, rc: 8 2020-12-29 02:40:20 [4122150.183730] Lustre: 18519:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1609238415/real 1609238420]  req@ffff9724f4650900 x1683090791024000/t0(0) o41-&gt;lsh-MDT000c-osp-MDT0005@172.19.3.13@o2ib600:24/4 lens 224/368 e 0 to 1 dl 1609238521 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 2020-12-29 02:40:20 [4122150.189175] Lustre: lsh-MDT0005: Received new LWP connection from 172.19.3.13@o2ib600, removing former export from same NID 2020-12-29 02:40:20 [4122150.189193] Lustre: lsh-MDT0005: Connection restored to  (at 172.19.3.13@o2ib600) 2020-12-29 02:40:20 [4122150.237218] Lustre: lsh-MDT000c-osp-MDT0005: Connection to lsh-MDT000c (at 172.19.3.13@o2ib600) was lost; in progress operations using this service will wait for recovery to complete</pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2020-12-29 07:27:28 [4139376.886078] LNetError: 12879:0:(o2iblnd_cb.c:3359:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds 2020-12-29 07:27:28 [4139376.897645] LNetError: 12879:0:(o2iblnd_cb.c:3434:kiblnd_check_conns()) Timed out RDMA with 172.19.3.13@o2ib600 (11): c: 7, oc: 0, rc: 8 2020-12-29 07:27:28 [4139376.912177] Lustre: 18530:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1609255643/real 1609255648]  req@ffff9760e9b92400 x1683090802516928/t0(0) o41-&gt;lsh-MDT000c-osp-MDT0005@172.19.3.13@o2ib600:24/4 lens 224/368 e 0 to 1 dl 1609255749 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 2020-12-29 07:27:28 [4139376.917039] Lustre: lsh-MDT0005: Received new LWP connection from 172.19.3.13@o2ib600, removing former export from same NID 2020-12-29 07:27:28 [4139376.917054] Lustre: lsh-MDT0005: Connection restored to  (at 172.19.3.13@o2ib600) 2020-12-29 07:27:28 [4139376.965671] Lustre: lsh-MDT000c-osp-MDT0005: Connection to lsh-MDT000c (at 172.19.3.13@o2ib600) was lost; in progress operations using this service will wait for recovery to complete </pre> </div></div> <p>and then timed out requests</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2020-12-29 09:49:57 [4147924.759391] Lustre: 18536:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1609264091/real 1609264091]  req@ffff9724efdeb180 x1683090808250432/t0(0) o400-&gt;lsh-MDT000c-osp-MDT0005@172.19.3.14@o2ib600:24/4 lens 224/224 e 0 to 1 dl 1609264197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 2020-12-29 09:50:22 [4147949.701554] Lustre: 18531:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1609264116/real 1609264116]  req@ffff9724efdebf00 x1683090808267264/t0(0) o400-&gt;lsh-MDT000c-osp-MDT0005@172.19.3.14@o2ib600:24/4 lens 224/224 e 0 to 1 dl 1609264222 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 </pre> </div></div> <p>Possibly related to <a href="https://jira.whamcloud.com/browse/LU-12735" title="MGS misbehaving in 2.12.2+" class="issue-link" data-issue-key="LU-12735"><del>LU-12735</del></a></p> <p>I have console logs for all 3 nodes.</p> <p>I do not have debug logs from the dates/times above because those entries were pushed out by newer ones.</p> <p>The problem seems to have persisted until I crashed the nodes today. I do have those crash dumps for the nodes with MGS,MDT0 and MDT5 in case there's useful information to be obtained there. </p> <p>Mike</p> <p>Could you please look into the reason behind the initial eviction here?</p> <p>Thanks</p> <p>Peter</p> <p>Hello,</p> <p>We experienced the same type of issue recently on a Lustre 2.12.4 MGS:</p> <p>After re-mounting an OST, the MGS display the same type of message, firstly for a EX lock and then for a CR lock on the same MGS's fs resources (LDLM_PLAIN, type=CONFIG_T_RECOVER).</p> <p>Then the all the MGCs disconnect/reconnect to the MGS every 5 minutes.</p> <p>After 1h30 the HA detect a connection issue (not responding) with the MGS's node and kill it (stonith).</p> <p>After some research we found that our issue could be associated with the following tickets: <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a>, <a href="https://jira.whamcloud.com/browse/LU-10674" title="MGS very unstable in 2.10.x" class="issue-link" data-issue-key="LU-10674">LU-10674</a> or <a href="https://jira.whamcloud.com/browse/LU-12735" title="MGS misbehaving in 2.12.2+" class="issue-link" data-issue-key="LU-12735"><del>LU-12735</del></a> tickets.</p> <p>Etienne, does your <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a> patch help with that issue? So far I'd say that is what needed</p> <p>I think so but I don't know for sure: there are no reproducer or tests.</p> <p>We are determining  causes of the MGS's failover from a crashdump. We think it might be that the mgs threads took too much CPU time (as in <a href="https://jira.whamcloud.com/browse/LU-12735" title="MGS misbehaving in 2.12.2+" class="issue-link" data-issue-key="LU-12735"><del>LU-12735</del></a>).</p> <p>In dmesg we have a message like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [3653076.838597] sched: RT throttling activated </pre> </div></div> <p>In vmcore, we see an accumulation of locks in the resource waiting list and nothing in the granted list (as say in a comment of the <a href="https://jira.whamcloud.com/browse/LU-10674" title="MGS very unstable in 2.10.x" class="issue-link" data-issue-key="LU-10674">LU-10674</a>'s patch).</p> Related LU-14379 Development Rank 1|i01kkf: Rank (Obsolete) 9223372036854775807 Severity