https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16235 Lustre <p>When adding new HSM actions llog records in mdt_agent_record_add(), cdt_state might be in CDT_INIT, so the HSM actions llog may not have been fully processed to set cdt_last_cookie to an appropriately large value, leading to cookie values being reused and triggering the assertions in cdt_agent_record_hash_add(). <a href="https://jira.whamcloud.com/browse/LU-13689" title="Replace cdt_state_lock with cdt_llog_lock" class="issue-link" data-issue-key="LU-13689">LU-13689</a> attempted to fix this, but there might be a simpler solution.</p> LU-16235

cdt_agent_record_hash_add() ASSERTION( carl0->carl_cat_idx == carl1->carl_cat_idx ) failed

Bug Minor In Progress Unresolved Nikitas Angelinas Nikitas Angelinas Wed, 12 Oct 2022 19:21:48 +0000 Fri, 12 Jan 2024 12:00:36 +0000 0 4 <p>"Nikitas Angelinas &lt;nikitas.angelinas@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48842" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48842</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16235" title="cdt_agent_record_hash_add() ASSERTION( carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx ) failed" class="issue-link" data-issue-key="LU-16235">LU-16235</a> hsm: check cdt_state before adding actions llog record<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 10fd63dfe8cd712abc848a72719b44c8759f85e5</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51850" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51850</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16235" title="cdt_agent_record_hash_add() ASSERTION( carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx ) failed" class="issue-link" data-issue-key="LU-16235">LU-16235</a> hsm: get a valid cookie for RAoLU request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 923f1081659826a10d3a10c43ed60453e934954c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48842/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48842/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16235" title="cdt_agent_record_hash_add() ASSERTION( carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx ) failed" class="issue-link" data-issue-key="LU-16235">LU-16235</a> hsm: check CDT state before adding actions llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fe5706e0c19f96e4f821790004f05ab265002e9d</p> <p>this patch causes GFP:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 9128.588763] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:32:30 (1693546350) ... [ 9196.856259] Lustre: Found index 0 <span class="code-keyword">for</span> lustre-MDT0000, updating log [ 9197.308569] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 9197.541121] Lustre: server umount lustre-MDT0000 complete [ 9197.920157] BUG: unable to handle kernel paging request at ffff89866f0e2698 [ 9197.920511] PGD 76e01067 P4D 76e01067 PUD 176f48067 PMD 176dcf067 PTE 800ffffed0f1d060 [ 9197.920558] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 9197.920586] CPU: 1 PID: 481978 Comm: hsm_cdtr Tainted: G W O --------- - - 4.18.0 #2 [ 9197.920636] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 9197.920690] RIP: 0010:mdt_coordinator+0xd7/0x1a10 [mdt] [ 9197.920728] Code: ff 01 db 74 31 31 db eb 0c 8b 05 7c 39 58 ff 01 c0 39 c3 73 21 bf 00 ca 9a 3b 83 c3 01 e8 01 81 28 e7 48 89 c7 e8 59 8a 71 e7 &lt;49&gt; 8b 84 24 98 06 00 00 a8 01 74 d3 49 8b 84 24 98 06 00 00 a8 01 [ 9197.920826] RSP: 0000:ffff898681d17e00 EFLAGS: 00010282 [ 9197.920853] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [ 9197.920891] RDX: 0000000000000000 RSI: ffffffffa8117897 RDI: 0000000000000246 [ 9197.920936] RBP: ffff8985a1c14740 R08: 0000000000000000 R09: ffff8986b13e98c0 [ 9197.920974] R10: 0000000000000000 R11: 000000000000004f R12: ffff89866f0e2000 [ 9197.921012] R13: ffff89868a508380 R14: ffffffffc0e8eb10 R15: ffff898648cbc000 [ 9197.921051] FS: 0000000000000000(0000) GS:ffff8986b1200000(0000) knlGS:0000000000000000 [ 9197.921089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9197.921120] CR2: ffff89866f0e2698 CR3: 0000000139335000 CR4: 00000000000006a0 [ 9197.921161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9197.921208] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9197.921246] Call Trace: [ 9197.921265] ? _raw_spin_lock_irqsave+0x46/0x80 [ 9197.921302] ? finish_task_switch+0x1f1/0x280 [ 9197.921350] ? set_cdt_state+0x40/0x40 [mdt] [ 9197.921386] kthread+0x129/0x140 [ 9197.921415] ? kthread_flush_work_fn+0x10/0x10 [ 9197.921451] ret_from_fork+0x1f/0x30 [ 9197.921472] Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) zunicode(O) zzstd(O) zlua(O) zcommon(O) znvpair(O) zavl(O) icp(O) spl(O) [last unloaded: libcfs] [ 9197.921647] CR2: ffff89866f0e2698 [ 9197.921668] ---[ end trace a7d48f6687796264 ]--- </pre> </div></div> <p>here is bt:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PID: 481978 TASK: ffff89868a508380 CPU: 1 COMMAND: <span class="code-quote">"hsm_cdtr"</span> #0 [ffff898681d17c68] panic at ffffffffa80b9786 /tmp/kernel/kernel/panic.c: 299 #1 [ffff898681d17d00] no_context at ffffffffa80a9563 /tmp/kernel/arch/x86/mm/fault.c: 799 #2 [ffff898681d17d50] page_fault at ffffffffa8600f0e /tmp/kernel/arch/x86/entry/entry_64.S: 1220 [exception RIP: mdt_coordinator+215] RIP: ffffffffc0e8ebe7 RSP: ffff898681d17e00 RFLAGS: 00010282 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffffa8117897 RDI: 0000000000000246 RBP: ffff8985a1c14740 R8: 0000000000000000 R9: ffff8986b13e98c0 R10: 0000000000000000 R11: 000000000000004f R12: ffff89866f0e2000 R13: ffff89868a508380 R14: ffffffffc0e8eb10 R15: ffff898648cbc000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 /home/lustre/linux-4.18.0-305.25.1.el8_4/./arch/x86/include/asm/bitops.h: 324 #3 [ffff898681d17f10] kthread at ffffffffa80d5199 /tmp/kernel/kernel/kthread.c: 340 #4 [ffff898681d17f50] ret_from_fork at ffffffffa860019f /tmp/kernel/arch/x86/entry/entry_64.S: 325 </pre> </div></div> <p>GFP was hit at:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">while</span> (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state) &amp;&amp; i &lt; obd_timeout) { </pre> </div></div> <p>this helped:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/mdt/mdt_coordinator.c b/lustre/mdt/mdt_coordinator.c index 439e0cc130..90f2c270df 100644 --- a/lustre/mdt/mdt_coordinator.c +++ b/lustre/mdt/mdt_coordinator.c @@ -605,6 +605,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> mdt_coordinator(void *data) cdt_start_pending_restore(mdt, cdt); set_cdt_state(cdt, CDT_RUNNING); + wake_up(&amp;cdt-&gt;cdt_waitq); <span class="code-keyword">while</span> (1) { <span class="code-object">int</span> i; @@ -1227,6 +1228,7 @@ <span class="code-object">int</span> mdt_hsm_cdt_stop(struct mdt_device *mdt) <span class="code-object">int</span> rc; ENTRY; + wait_event(cdt-&gt;cdt_waitq, cdt-&gt;cdt_state != CDT_INIT); <span class="code-comment">/* stop coordinator thread */</span> rc = set_cdt_state(cdt, CDT_STOPPING); <span class="code-keyword">if</span> (rc == 0) { </pre> </div></div> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52222" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52222</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-16235" title="cdt_agent_record_hash_add() ASSERTION( carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx ) failed" class="issue-link" data-issue-key="LU-16235">LU-16235</a> hsm: check CDT state before adding actions llog"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: da1da220fe5176331145bc967d0b1a86f8f26433</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=eaujames" class="user-hover" rel="eaujames">eaujames</a>, could you please see some questions re if we need the revert patch in <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/52222?" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/52222?</a></p> <p>I have abandoned the revert and merge it with the <a href="https://review.whamcloud.com/51256" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/51256</a> ("<a href="https://jira.whamcloud.com/browse/LU-16356" title="high contention on cdt_request_lock causes clients to hang" class="issue-link" data-issue-key="LU-16356">LU-16356</a> hsm: add running ref to the coordinator ")</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53660" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53660</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16235" title="cdt_agent_record_hash_add() ASSERTION( carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx ) failed" class="issue-link" data-issue-key="LU-16235">LU-16235</a> hsm: check CDT state before adding actions llog<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 2217ee7e5b18aee5fe0c9d6135194487afa38db5</p> Duplicate LU-13689 Related LU-16356 Development Rank 1|i032qv: Rank (Obsolete) 9223372036854775807 Severity