https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14474 Lustre <p><a href="https://testing.whamcloud.com/test_sets/fd169bf5-481b-4906-8891-b5fc6738d14d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/fd169bf5-481b-4906-8891-b5fc6738d14d</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [28530.564627] general protection fault: 0000 [#1] SMP PTI [28530.565704] CPU: 0 PID: 1366058 Comm: mdt00_002 Kdump: loaded Tainted: P OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [28530.567756] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [28530.568756] RIP: 0010:up_write+0x5/0x30 [28530.569406] Code: f8 02 74 01 c3 48 8b 47 20 a8 04 74 06 f0 48 83 67 20 fb e9 3d ff ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 &lt;48&gt; c7 47 20 00 00 00 00 48 c7 c0 ff ff ff ff f0 48 0f c1 07 a8 02 [28530.572467] RSP: 0018:ffffab6000b93a38 EFLAGS: 00010212 [28530.573333] RAX: 00000000ffffffea RBX: ffff97eb10f06a00 RCX: 000000000293cb3b [28530.574512] RDX: 000000000293cb3a RSI: 0000000000000002 RDI: 5f32303030646f6c [28530.575677] RBP: ffff97eb10f06ac8 R08: 000000000002f0c0 R09: ffffffffc1a13f21 [28530.576846] R10: 00000000ffffffea R11: 0000000000000001 R12: 0000000000000001 [28530.578021] R13: ffff97eb0f0ee140 R14: ffff97eb2d2f7cb8 R15: 00000000ffffffea [28530.579186] FS: 0000000000000000(0000) GS:ffff97eb3fc00000(0000) knlGS:0000000000000000 [28530.580510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [28530.581457] CR2: 00007fcd6fb50000 CR3: 0000000042a0a005 CR4: 00000000000606f0 [28530.582640] Call Trace: [28530.583369] llog_cat_prep_log+0xb0/0x3c0 [obdclass] [28530.584256] llog_cat_declare_add_rec+0xbe/0x220 [obdclass] [28530.585211] llog_declare_add+0x187/0x1d0 [obdclass] [28530.586470] top_trans_start+0x212/0x940 [ptlrpc] [28530.587364] mdd_create+0xba4/0x1850 [mdd] [28530.588225] mdt_create+0x7b1/0xc50 [mdt] [28530.588936] mdt_reint_create+0x30b/0x3c0 [mdt] [28530.589721] mdt_reint_rec+0x11f/0x250 [mdt] [28530.590454] mdt_reint_internal+0x498/0x780 [mdt] [28530.591283] mdt_reint+0x5e/0x100 [mdt] [28530.591981] tgt_request_handle+0xc78/0x1910 [ptlrpc] </pre> </div></div> LU-14474

Oops in llog_cat_prep_log() in sanity-quota / recovery-small

Bug Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev zfs Sat, 27 Feb 2021 04:51:08 +0000 Thu, 15 Dec 2022 21:04:53 +0000 Sun, 10 Oct 2021 15:16:40 +0000 Lustre 2.15.0 0 3 <p>+1 on master in recovery-small unmount <a href="https://testing.whamcloud.com/test_sets/e3bb9ef5-9a74-4037-9fd5-bd9c1e60e4e2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e3bb9ef5-9a74-4037-9fd5-bd9c1e60e4e2</a></p> <p>+1 on master in recovery-small after the test had finished:<br/> <a href="https://testing.whamcloud.com/test_sets/6eb8a766-eea5-44d8-8fcb-0eac77b2be79" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6eb8a766-eea5-44d8-8fcb-0eac77b2be79</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == recovery-small test complete, duration 6212 sec ======= 13:58:23 (1629640703) BUG: unable to handle kernel paging request at 000000000bd00c23 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 2618 Comm: mdt00_001 Kdump: loaded 4.18.0-240.22.1.el8_lustre.x86_64 #1 RIP: 0010:llog_exist+0xd9/0x180 [obdclass] Call Trace: llog_cat_prep_log+0x4f/0x3c0 [obdclass] llog_cat_declare_add_rec+0x56/0x220 [obdclass] llog_declare_add+0x187/0x1d0 [obdclass] top_trans_start+0x212/0x940 [ptlrpc] mdd_unlink+0x4a0/0xb30 [mdd] mdt_reint_unlink+0xb0c/0x1290 [mdt] mdt_reint_rec+0x11f/0x250 [mdt] mdt_reint_internal+0x498/0x780 [mdt] mdt_reint+0x5e/0x100 [mdt] tgt_request_handle+0xc90/0x1940 [ptlrpc] ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc] ptlrpc_main+0xba2/0x1490 [ptlrpc] </pre> </div></div> <p>a similar one: <a href="https://testing.whamcloud.com/test_sets/a3d67622-925a-4122-abf5-f599b59970ba" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a3d67622-925a-4122-abf5-f599b59970ba</a></p> <p><a href="https://testing.whamcloud.com/test_sessions/66dd0fd7-ab82-4af5-9178-0a12fe5302d5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/66dd0fd7-ab82-4af5-9178-0a12fe5302d5</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 857.600721] LustreError: 167-0: lustre-MDT0001-osp-MDT0002: This client was evicted by lustre-MDT0001; in progress operations using <span class="code-keyword">this</span> service will fail. [ 857.857771] LustreError: 64216:0:(llog.c:472:llog_verify_record()) lustre-MDT0001-osp-MDT0000: record is too large: 0 &gt; 32768 [ 857.859905] LustreError: 64216:0:(llog.c:656:llog_process_thread()) lustre-MDT0001-osp-MDT0000: invalid record in llog [0x1:0x40001b70:0x2] record <span class="code-keyword">for</span> index 0/1: rc = -22 </pre> </div></div> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44910" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44910</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14474" title="Oops in llog_cat_prep_log() in sanity-quota / recovery-small" class="issue-link" data-issue-key="LU-14474"><del>LU-14474</del></a> obdclass: debugging bits<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fbbd7933cc997754f874ad8ec964da8ea1e7df72</p> <p>the immediate reason is that lod_sub_recovery_thread() destroys just scanned llog which is stored in chd_next_log, but I haven't understood yet how the log destroyed got into chd_next_log.<br/> one thing I observed is that llog processing in lod_sub_recovery_thread() races with <em>new</em> requests from the clients. I guess this is not quite correct.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44998" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44998</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14474" title="Oops in llog_cat_prep_log() in sanity-quota / recovery-small" class="issue-link" data-issue-key="LU-14474"><del>LU-14474</del></a> llog: reset pointer to the next llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 447f84ff513e2ab3efdac5b32403ac06d3c61e7c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44998/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44998/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14474" title="Oops in llog_cat_prep_log() in sanity-quota / recovery-small" class="issue-link" data-issue-key="LU-14474"><del>LU-14474</del></a> llog: don't destroy next llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4521f6af35d1dc20b531b87ff3633d89dbac86ec</p> <p>Landed for 2.15</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45928" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45928</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14474" title="Oops in llog_cat_prep_log() in sanity-quota / recovery-small" class="issue-link" data-issue-key="LU-14474"><del>LU-14474</del></a> llog: don't destroy next llog<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 61ca609907943a56e94e073ee0d71ead0074aeb5</p> Related LU-14964 LU-14932 LU-14995 LU-16398 Development Rank 1|i01nx3: Rank (Obsolete) 9223372036854775807 Severity