https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12212 Lustre <p>Ordinary <tt>dbench</tt> run start showing a lot of messages like this:</p> <blockquote> <p>Apr 21 03:11:05 nodez kernel: Lustre: 4236:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1555830658/real 1555830658&#93;</span> req@ffff8800a273cc00 x1631406558218272/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff8800af2fb800@0@lo:12/10 lens 616/4752 e 0 to 1 dl 1555830665 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 21 03:11:05 nodez kernel: Lustre: lustre-MDT0000-mdc-ffff8800af2fb800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete<br/> Apr 21 03:11:05 nodez kernel: Lustre: lustre-MDT0000: Client 2f22eecb-5055-b4d6-16b5-a958700fcbda (at 0@lo) reconnecting<br/> Apr 21 03:11:05 nodez kernel: Lustre: lustre-MDT0000: Connection restored to 4e8c3a45-3b8a-cd6a-047f-22363e8171e6 (at 0@lo)<br/> Apr 21 03:11:05 nodez kernel: Lustre: Skipped 3 previous similar messages<br/> Apr 21 03:11:48 nodez kernel: Lustre: 4237:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1555830701/real 1555830701&#93;</span> req@ffff8800862f3440 x1631406567790080/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff8800af2fb800@0@lo:12/10 lens 616/4752 e 0 to 1 dl 1555830708 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 21 03:11:48 nodez kernel: Lustre: lustre-MDT0000-mdc-ffff8800af2fb800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete<br/> Apr 21 03:11:48 nodez kernel: Lustre: lustre-MDT0000: Client 2f22eecb-5055-b4d6-16b5-a958700fcbda (at 0@lo) reconnecting<br/> Apr 21 03:12:12 nodez kernel: Lustre: 4238:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1555830725/real 1555830725&#93;</span> req@ffff8800861750c0 x1631406572864272/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff8800af2fb800@0@lo:12/10 lens 616/4752 e 0 to 1 dl 1555830732 ref 2 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 21 03:12:12 nodez kernel: Lustre: lustre-MDT0000-mdc-ffff8800af2fb800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete<br/> Apr 21 03:12:12 nodez kernel: Lustre: lustre-MDT0000: Client 2f22eecb-5055-b4d6-16b5-a958700fcbda (at 0@lo) reconnecting<br/> Apr 21 03:12:12 nodez kernel: Lustre: lustre-MDT0000: Connection restored to 4e8c3a45-3b8a-cd6a-047f-22363e8171e6 (at 0@lo)<br/> Apr 21 03:12:12 nodez kernel: Lustre: Skipped 3 previous similar messages</p></blockquote> <p>This started after <a href="https://jira.whamcloud.com/browse/LU-9193" title="Multiple hangs observed with many open/getattr" class="issue-link" data-issue-key="LU-9193"><del>LU-9193</del></a> patch landing (found by <tt>git bisect</tt>). There is nothing special with test setup, no SELinux, just local run with <tt>dbench -D /mnt/lustre/testdir 4</tt></p> LU-12212

Often requests timeouts during dbench run

Bug Critical Resolved Fixed Mikhail Pershin Mikhail Pershin LTS12 Sun, 21 Apr 2019 07:22:45 +0000 Mon, 1 Feb 2021 18:08:38 +0000 Tue, 21 May 2019 14:33:09 +0000 Lustre 2.13.0 0 5 <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34734" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34734</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12212" title="Often requests timeouts during dbench run" class="issue-link" data-issue-key="LU-12212"><del>LU-12212</del></a> mdt: fix SECCTX reply buffer handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 63a53d131de12c61657d4299530a5eafcb7dab8e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34734/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34734/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12212" title="Often requests timeouts during dbench run" class="issue-link" data-issue-key="LU-12212"><del>LU-12212</del></a> mdt: fix SECCTX reply buffer handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cb61ed93f8563c26b6a6db396478fe54f8dc42cb</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34910" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34910</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12212" title="Often requests timeouts during dbench run" class="issue-link" data-issue-key="LU-12212"><del>LU-12212</del></a> mdt: fix SECCTX reply buffer handling<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 68f7f348553c5d55a14165a6227a9a40756dfb50</p> <p>Is there a reason not to re-apply <a href="https://jira.whamcloud.com/browse/LU-9193" title="Multiple hangs observed with many open/getattr" class="issue-link" data-issue-key="LU-9193"><del>LU-9193</del></a> on b2_12 with this patch on top of it ?</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-9193" title="Multiple hangs observed with many open/getattr" class="issue-link" data-issue-key="LU-9193"><del>LU-9193</del></a> is a very long-standing issue for several of our customers, and we would rather not wait for the next LTS to have it solved.</p> Related LU-9193 Development Rank 1|i00f5z: Rank (Obsolete) 9223372036854775807 Severity