https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16539 Lustre <p>This issue was created by maloo for Timothy Day &lt;timday@amazon.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5ae4d92c-fc89-4e8c-bf83-4b09c9e7ead2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5ae4d92c-fc89-4e8c-bf83-4b09c9e7ead2</a></p> <p>test_32 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 132 minutes, last suite running was replay-dual </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/92186" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92186</a> - 4.18.0-372.32.1.el8_6.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/92186" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92186</a> - 4.18.0-372.32.1.el8_lustre.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>Maloo auto-associated EX-6786 (this isn't community viewable, but I it put here for the benefit of those that can see it).</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-dual test_32 - Timeout occurred after 132 minutes, last suite running was replay-dual</p> LU-16539

replay-dual test_32: Timeout occurred after x minutes, last suite running was replay-dual

Bug Minor Open Unresolved WC Triage Maloo Wed, 8 Feb 2023 16:36:33 +0000 Wed, 14 Jun 2023 00:01:20 +0000 0 5 <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/c9a4a588-8f2d-4d3e-8129-33b0f89ad07a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c9a4a588-8f2d-4d3e-8129-33b0f89ad07a</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/40925114-446c-4b59-a1d2-001ad058c500" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/40925114-446c-4b59-a1d2-001ad058c500</a></p> <p>The 'ost' service seems to hang:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Mon Jun 5 11:32:14 2023] Lustre: DEBUG MARKER: == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 11:32:14 (1685964734) [Mon Jun 5 11:32:30 2023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.240.25.55@tcp) was lost; in progress operations using this service will wait for recovery to complete [Mon Jun 5 11:32:30 2023] Lustre: Skipped 71 previous similar messages [Mon Jun 5 11:32:35 2023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-61vm4.onyx.whamcloud.com: executing set_default_debug -1 all 4 [Mon Jun 5 11:32:35 2023] LustreError: 166-1: MGC10.240.25.55@tcp: Connection to MGS (at 10.240.25.55@tcp) was lost; in progress operations using this service will fail [Mon Jun 5 11:32:35 2023] LustreError: Skipped 4 previous similar messages [Mon Jun 5 11:38:00 2023] LustreError: 94760:0:(ldlm_request.c:124:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1685964753, 328s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0004_UUID lock: 000000002d1bbe55/0xec7df65557708c7c lrc: 3/0,1 mode: --/PW res: [0x400000404:0x5c4:0x0].0x0 rrc: 5 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;18446744073709551615) gid 0 flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 94760 timeout: 0 lvb_type: 0 [Mon Jun 5 11:38:00 2023] LustreError: 93962:0:(ldlm_request.c:124:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [Mon Jun 5 11:39:05 2023] ptlrpc_watchdog_fire: 69 callbacks suppressed [Mon Jun 5 11:39:05 2023] Lustre: ll_ost00_068: service thread pid 93962 was inactive for 393.001 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Mon Jun 5 11:39:05 2023] Pid: 94760, comm: ll_ost00_074 4.18.0-425.10.1.el8_lustre.x86_64 #1 SMP Wed May 3 16:22:26 UTC 2023 [Mon Jun 5 11:39:05 2023] Call Trace TBD: [Mon Jun 5 11:39:05 2023] Lustre: Skipped 2 previous similar messages [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ldlm_completion_ast+0x57e/0x900 [ptlrpc] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ldlm_cli_enqueue_local+0x30b/0x860 [ptlrpc] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ofd_destroy_by_fid+0x239/0x4a0 [ofd] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ofd_destroy_hdl+0x263/0xa00 [ofd] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] tgt_request_handle+0xcc3/0x1920 [ptlrpc] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ptlrpc_server_handle_request+0x31d/0xbc0 [ptlrpc] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ptlrpc_main+0xc52/0x1510 [ptlrpc] [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] kthread+0x10b/0x130 [Mon Jun 5 11:39:05 2023] [&lt;0&gt;] ret_from_fork+0x35/0x40 [Mon Jun 5 12:26:36 2023] Lustre: 'ost' is processing requests too slowly, client may timeout. Late by 13s, missed 0 early replies (reqs waiting=0 active=78, at_estimate=600, delay=0ms) </pre> </div></div> <p>Failed 10x on master in the past week.</p> Duplicate Related LU-15809 Development Rank 1|i03cc7: Rank (Obsolete) 9223372036854775807 Severity