Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-12741] crash in osd_object_delete at end of sanity https://jira.whamcloud.com/browse/LU-12741 Lustre <p>It looks like something broke in master/b2_12 relatively recently.</p> <p>Typical crash:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[14473.601088] Lustre: DEBUG MARKER: == sanity test complete, duration 5433 sec =========================================================== 03:14:06 (1567926846) [14492.366277] BUG: unable to handle kernel NULL pointer dereference at 0000000000000c80 [14492.370164] IP: [&lt;ffffffffa0b8e854&gt;] osd_object_delete+0x1f4/0x2a0 [osd_ldiskfs] [14492.372007] PGD 0 [14492.372787] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [14492.373666] Modules linked in: dm_flakey dm_mod lustre(OE) mdt(OE) mdd(OE) mdc(OE) obdecho(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) mgc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) brd ext4 loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon binfmt_misc ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata floppy virtio_blk serio_raw i2c_core [last unloaded: mdt] [14492.388104] CPU: 6 PID: 8302 Comm: ldlm_cn03_003 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.6-debug #2 [14492.389885] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [14492.390901] task: ffff88003c6209c0 ti: ffff88004139c000 task.ti: ffff88004139c000 [14492.392652] RIP: 0010:[&lt;ffffffffa0b8e854&gt;] [&lt;ffffffffa0b8e854&gt;] osd_object_delete+0x1f4/0x2a0 [osd_ldiskfs] [14492.394473] RSP: 0018:ffff88004139fa30 EFLAGS: 00010246 [14492.395341] RAX: 0000000000000000 RBX: 0000000000000c80 RCX: 0000000000000000 [14492.396537] RDX: ffff88006430ae00 RSI: ffffffffa0be0160 RDI: ffff880082322b40 [14492.397291] RBP: ffff88004139fa60 R08: 0000000000000000 R09: d8c8000000000000 [14492.398095] R10: ffff8800bb75e000 R11: ffff8800bb75e7c8 R12: 0000000000000000 [14492.398993] R13: ffff880115978e00 R14: ffff880082322b40 R15: 0000000000000000 [14492.399675] FS: 0000000000000000(0000) GS:ffff880139780000(0000) knlGS:0000000000000000 [14492.401123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14492.402167] CR2: 0000000000000c80 CR3: 0000000130b34000 CR4: 00000000000006e0 [14492.422237] Call Trace: [14492.423115] [&lt;ffffffffa036d0a5&gt;] lu_object_free.isra.31+0x65/0x170 [obdclass] [14492.424986] [&lt;ffffffffa0370e42&gt;] lu_object_put+0xc2/0x3c0 [obdclass] [14492.426016] [&lt;ffffffffa0d7fea0&gt;] ? mdt_punch_hpreq_fini+0x10/0x10 [mdt] [14492.427067] [&lt;ffffffffa0d7ff51&gt;] ldlm_dom_discard_cp_ast+0xb1/0x2b0 [mdt] [14492.428115] [&lt;ffffffffa05f25c6&gt;] ldlm_work_cp_ast_lock+0xa6/0x1d0 [ptlrpc] [14492.429196] [&lt;ffffffffa06393b0&gt;] ptlrpc_set_wait+0x70/0x790 [ptlrpc] [14492.430253] [&lt;ffffffffa062fe6d&gt;] ? ptlrpc_prep_set+0x5d/0x290 [ptlrpc] [14492.431457] [&lt;ffffffffa0350279&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [14492.432481] [&lt;ffffffff810b5a70&gt;] ? __init_waitqueue_head+0x20/0x30 [14492.433523] [&lt;ffffffffa062ff07&gt;] ? ptlrpc_prep_set+0xf7/0x290 [ptlrpc] [14492.434526] [&lt;ffffffffa05f7e15&gt;] ldlm_run_ast_work+0xd5/0x380 [ptlrpc] [14492.435592] [&lt;ffffffffa05f927f&gt;] __ldlm_reprocess_all+0xff/0x340 [ptlrpc] [14492.436653] [&lt;ffffffffa05f94d0&gt;] ldlm_reprocess_all+0x10/0x20 [ptlrpc] [14492.437708] [&lt;ffffffffa06219b4&gt;] ldlm_handle_convert0+0x2f4/0x450 [ptlrpc] [14492.438727] [&lt;ffffffffa0621ffb&gt;] ldlm_cancel_handler+0x29b/0x590 [ptlrpc] [14492.439812] [&lt;ffffffffa06524b6&gt;] ptlrpc_server_handle_request+0x256/0xad0 [ptlrpc] [14492.441715] [&lt;ffffffffa06564a1&gt;] ptlrpc_main+0xb91/0x2110 [ptlrpc] [14492.442734] [&lt;ffffffff810c32ed&gt;] ? finish_task_switch+0x5d/0x1b0 [14492.443744] [&lt;ffffffff817b6cd0&gt;] ? __schedule+0x410/0xa00 [14492.445222] [&lt;ffffffffa0655910&gt;] ? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc] [14492.447031] [&lt;ffffffff810b4ed4&gt;] kthread+0xe4/0xf0 [14492.447974] [&lt;ffffffff810b4df0&gt;] ? kthread_create_on_node+0x140/0x140 [14492.448960] [&lt;ffffffff817c4c77&gt;] ret_from_fork_nospec_begin+0x21/0x21 [14492.449957] [&lt;ffffffff810b4df0&gt;] ? kthread_create_on_node+0x140/0x140 [14492.450934] Code: e0 03 00 0f 1f 40 00 4d 85 ed 74 c0 4c 89 f7 48 c7 c6 60 01 be a0 e8 cc e7 7d ff 49 89 c4 44 89 f8 31 c9 49 8d 9c 24 80 0c 00 00 &lt;49&gt; 89 84 24 80 0c 00 00 4c 89 ee 4c 89 f7 48 89 da e8 86 16 f0 </pre> </div></div> <p>Always in dom discard path</p> <p>First master-next occurence in githash f8c100f.</p> <p>List of patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>f8c100f LU-12575 build: add ibutils2 for MOFED build 0ecb29f LU-12560 tests: Use full path for test-groups 4f4f90b LU-12400 ptlrpc: Sun RPC changes for RCU locking 5a19817 LU-12527 utils: Make lustre_user.h c++-legal f0ba5de LU-12472 tests: update sanity-krb5.sh adfa543 LU-4315 doc: split lctl get_param and set_param man pages a57ede6 LU-12355 ldiskfs: Remove old map blocks support 87f6b68 LU-12405 lnet: Oracle OFED extensions default to on 04cdd15 LU-12343 osc: Fix dom handling in weight_ast 3d1920a LU-8066 mdt: migrate procfs files to sysfs bc02a4e LU-12075 mdt: commit migrate transaction with locks held 27cd9fd LU-10070 test: llapi_layout_test enhancements 2490ed4 LU-11617 mdc: fix possible deadlock in chlg_open() 860dbcb LU-12559 ptlrpc: Hold imp lock for idle reconnect ce3ccbd LU-6142 tests: Fix style issues for write_disjoint.c 6012e3e LU-6142 tests: Fix style issues for write_append_truncate.c 16792c9 LU-6142 tests: Fix style issues for lp_utils.c ac153a9 LU-10094 mdc: dir page ldp_hash_end mistakenly adjusted b598d82 LU-12523 ptlrpc: Don't get jobid in body_v2 b2f2bfc LU-6202 utils: remove obsolete l_ioctl2() wrapper 42fdd2f LU-12440 lnet: Misleading error from lnet_is_health_check aec7b1a LU-12439 lnet: Convert noisy timeout error to cdebug 93419c4 LU-11023 quota: remove quota pool ID </pre> </div></div> <p>First b2_12-next occurrence githash: 8ec5896 list of patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>3674d393d5 LU-12608 kernel: kernel update RHEL7.6 [3.10.0-957.27.2.el7] fe03ca414f LU-11761 fld: let's caller to retry FLD_QUERY 316310cbb4 LU-12387 tests: Validate l_tunedisk max_sectors_kb tuning 0edb0a6951 LU-8130 libcfs: don't include rhashtable if unavailable 9ac11632fb LU-12660 kernel: kernel update SLES12 SP4 [4.12.14-95.29.1] 3a35d97aee LU-12539 build: pass --with-o2ib when building deb packages e2ac8c3269 LU-10094 mdc: dir page ldp_hash_end mistakenly adjusted cbb6d8c8ef LU-12586 lov: Correct write_intent end for trunc a1e888dcbc LU-10756 ptlrpc: change IMPORT_SET_* macros into real functions bbf40d8c71 LU-11537 osp: avoid nested transaction 61c93c46c4 LU-12343 osc: Fix dom handling in weight_ast </pre> </div></div> <p>I guess <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a> is the common factor here?</p> LU-12741 crash in osd_object_delete at end of sanity Bug Major Resolved Fixed Mikhail Pershin Oleg Drokin Tue, 10 Sep 2019 08:11:44 +0000 Thu, 24 Sep 2020 09:33:18 +0000 Sat, 14 Dec 2019 13:29:04 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.4 0 6 <p>Oleg,</p> <p>Can you provide some links to your crash/test failure stuff from your rig?</p> <p>Hmm, Oleg, this <b>really</b> looks like it should be linked to <a href="https://jira.whamcloud.com/browse/LU-11359" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11359</a></p> <p>rather than <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a>.</p> <p>How sure are you it started with <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a>...?</p> <p>This can be related to <a href="https://jira.whamcloud.com/browse/LU-11359" title="racer test 1 times out with client hung in dir_create.sh, ls, … and MDS in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-11359"><del>LU-11359</del></a> but that issue started to show up only recently so something was changed by one of the recent patches, probably it is <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a> patch because it was landed on both b2_12 and master when this issue appeared. I don't think patch from <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a> is making something wrong but it can be trigger for some missed race in lu_object_put. I will re-check <a href="https://jira.whamcloud.com/browse/LU-11359" title="racer test 1 times out with client hung in dir_create.sh, ls, … and MDS in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-11359"><del>LU-11359</del></a> also.</p> <p>Mike,</p> <p>Do you want to take this one, then?  I was going to look at it but I'm happy to let you do it.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pfarrell" class="user-hover" rel="pfarrell">pfarrell</a> as you can see that <a href="https://jira.whamcloud.com/browse/LU-12343" title="osc_lock.c:687:osc_ldlm_weigh_ast()) LBUG" class="issue-link" data-issue-key="LU-12343"><del>LU-12343</del></a> is the common patch between the patchset where this failure appeared on master and when it appeared on b2_12 (different set of patches mostly)</p> <p>Also it's probably not a pure coincidence that your patch deals with DOM AST logic and the failure is in a DOM AST path.</p> <p>Mike is looking into this</p> <p>Seems to have stopped occurring on master. Likely a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11204" title="mdt_reint_unlink-&gt;lu_object_put() crash" class="issue-link" data-issue-key="LU-11204"><del>LU-11204</del></a></p> <p>actually still happens, fresh master:<br/> Lustre: DEBUG MARKER: == sanity test complete, duration 4910 sec =========================================================== 21:34:43 (1573144483)<br/> LustreError: 23081:0:(osd_handler.c:2142:osd_object_delete()) 000000008c85a7cf after refill<br/> LustreError: 23081:0:(osd_handler.c:2143:osd_object_delete()) LBUG<br/> lbug_with_loc+0x79/0x80 <span class="error">&#91;libcfs&#93;</span><br/> ? osd_object_delete+0x319/0x320 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> ? lu_object_free.isra.0+0x44/0x140 <span class="error">&#91;obdclass&#93;</span><br/> ? lu_object_put+0x230/0x370 <span class="error">&#91;obdclass&#93;</span><br/> ? mdt_punch_hpreq_fini+0x10/0x10 <span class="error">&#91;mdt&#93;</span><br/> ? ldlm_dom_discard_cp_ast+0xe0/0x240 <span class="error">&#91;mdt&#93;</span><br/> ? ldlm_work_cp_ast_lock+0xec/0x1d0 <span class="error">&#91;ptlrpc&#93;</span><br/> ? ptlrpc_set_wait+0x4a/0x760 <span class="error">&#91;ptlrpc&#93;</span><br/> ? is_module_address+0xc/0x20<br/> ? static_obj+0x31/0x50<br/> ? __lockdep_init_map+0x45/0x180<br/> ? __raw_spin_lock_init+0x28/0x50<br/> ? ldlm_run_ast_work+0xbf/0x3b0 <span class="error">&#91;ptlrpc&#93;</span></p> <p>this is because of info=NULL, lu_env_refill() helps</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36714" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36714</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12741" title="crash in osd_object_delete at end of sanity" class="issue-link" data-issue-key="LU-12741"><del>LU-12741</del></a> ptlrpc: do lu_env_refill for new request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fb60e3076e1a5769e8051b2b268a7797e4f2dab5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36714/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36714/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12741" title="crash in osd_object_delete at end of sanity" class="issue-link" data-issue-key="LU-12741"><del>LU-12741</del></a> ptlrpc: do lu_env_refill for new request<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3f304b75d24aea0075415affa0c0bef004ef012c</p> <p>patch landed</p> <p>Reopen, still may require backporting</p> <p>Landed for 2.14. Backporting will be tracked separately.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37036" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37036</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12741" title="crash in osd_object_delete at end of sanity" class="issue-link" data-issue-key="LU-12741"><del>LU-12741</del></a> ptlrpc: do lu_env_refill for new request<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f6041a258b595d3c20bcd6a6994a1c284d094413</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37036/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37036/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12741" title="crash in osd_object_delete at end of sanity" class="issue-link" data-issue-key="LU-12741"><del>LU-12741</del></a> ptlrpc: do lu_env_refill for new request<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 0d79ba70e44f44d0334253f8c39725ba3d3f36e7</p> Duplicate LU-11204 Related LU-8992 LU-13980 Development Rank 1|i00mhr: Rank (Obsolete) 9223372036854775807 Severity