https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13508 Lustre <p>This seems to only trigger when running in "one unmount/mount per subtest" mode like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FLAKEY=false REFORMAT=yes ONLY="160-999" MDSCOUNT=2 SETUP=setup CLEANUP=cleanup sh sanity.sh </pre> </div></div> <p>The crash is 100% reproducible on a single node testing and looks like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[421223.630585] Lustre: server umount lustre-OST0001 complete [421223.879781] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC [421223.881119] Modules linked in: ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) pcc_cpufreq loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 binfmt_misc ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core floppy [last unloaded: lustre] [421223.890017] CPU: 1 PID: 18476 Comm: sh Kdump: loaded Tainted: P OE ------------ 3.10.0-7.7-debug #1 [421223.890922] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [421223.891370] task: ffff8803b185a340 ti: ffff8800ca418000 task.ti: ffff8800ca418000 [421223.892207] RIP: 0010:[&lt;ffffffff8110a016&gt;] [&lt;ffffffff8110a016&gt;] module_put+0x26/0xc0 [421223.893060] RSP: 0018:ffff8800ca41bda8 EFLAGS: 00010202 [421223.893504] RAX: ffff8800ca41bfd8 RBX: 6b6b6b6b6b6b6b6b RCX: ffff8800ca41bce0 [421223.894355] RDX: 000000000000000b RSI: 800000038530b060 RDI: 6b6b6b6b6b6b6b6b [421223.895185] RBP: ffff8800ca41bdc0 R08: 0000000000000001 R09: 0000000000000067 [421223.896016] R10: ffff880000000148 R11: 0000000000000000 R12: 6b6b6b6b6b6b6b6b [421223.896871] R13: ffff8803dd8d6d30 R14: ffff8803ca1ff168 R15: ffff8803d43ece60 [421223.897696] FS: 00007ffa9ed7f740(0000) GS:ffff8803f4640000(0000) knlGS:0000000000000000 [421223.898540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [421223.898991] CR2: 00007f119516c8d0 CR3: 0000000001c10000 CR4: 00000000000006e0 [421223.899845] Call Trace: [421223.900249] [&lt;ffffffff812448e0&gt;] cdev_put+0x20/0x30 [421223.900684] [&lt;ffffffff812415c6&gt;] __fput+0x266/0x300 [421223.901129] [&lt;ffffffff8124173e&gt;] ____fput+0xe/0x10 [421223.901565] [&lt;ffffffff810b4af5&gt;] task_work_run+0xb5/0xf0 [421223.902010] [&lt;ffffffff81092395&gt;] do_exit+0x2f5/0xb30 [421223.902450] [&lt;ffffffff817e0ee1&gt;] ? system_call_after_swapgs+0xae/0x146 [421223.902941] [&lt;ffffffff81092c4f&gt;] do_group_exit+0x3f/0xa0 [421223.904158] [&lt;ffffffff81092cc4&gt;] SyS_exit_group+0x14/0x20 [421223.904605] [&lt;ffffffff817e0f98&gt;] system_call_fastpath+0x1f/0x24 [421223.905071] [&lt;ffffffff817e0ee1&gt;] ? system_call_after_swapgs+0xae/0x146 [421223.905550] Code: 00 00 00 00 00 0f 1f 44 00 00 55 48 85 ff 48 89 e5 41 55 41 54 49 89 fc 53 74 3c 65 48 8b 04 25 f8 0e 01 00 83 80 44 c0 ff ff 01 &lt;48&gt; 8b 87 30 02 00 00 65 48 ff 40 08 4c 8b 6d 08 0f 1f 44 00 00 [421223.907311] RIP [&lt;ffffffff8110a016&gt;] module_put+0x26/0xc0 [421223.907761] RSP &lt;ffff8800ca41bda8&gt; </pre> </div></div> <p>I thin kit must be due to some recent module changes for upstreaming/cleanup reasons?</p> LU-13508

crash in sanity test 160j

Bug Critical Resolved Fixed Hongchao Zhang Oleg Drokin Sat, 2 May 2020 04:25:23 +0000 Fri, 19 Jun 2020 22:07:33 +0000 Fri, 19 Jun 2020 22:07:33 +0000 Lustre 2.14.0 Lustre 2.14.0 0 6 <p>The module pointer passed to module_put is 0x6b6b6b6b6b6b6b6b which suggests the structure storing the pointer has been freed - 6b is POISON_FREE.</p> <p>The pointer was in a 'struct cdev' (passed to cdev_put) and the only on in lustre is ced_cdev inside 'struct chlg_registered_dev(), so it seems one of those was freed too early.</p> <p>That seems to point to</p> <p>Commit d0423abc1adc ("<a href="https://jira.whamcloud.com/browse/LU-12506" title="Client unable to mount filesystem with very large number of MDTs" class="issue-link" data-issue-key="LU-12506"><del>LU-12506</del></a> changelog: support large number of MDT")</p> <p>I cannot immediately see the problem there.  Maybe I'll have another look later.</p> <p> </p> <p>That was a great guess, I reverted d0423abc1adc and sure enough the crash is gone.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hongchao.zhang" class="user-hover" rel="hongchao.zhang">hongchao.zhang</a> can you please take a look at this?</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38658" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38658</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13508" title="crash in sanity test 160j" class="issue-link" data-issue-key="LU-13508"><del>LU-13508</del></a> mdc: chlg device could be used after free<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2e8a101b2702da08dbd1e294732567dcad4a2660</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38658/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38658/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13508" title="crash in sanity test 160j" class="issue-link" data-issue-key="LU-13508"><del>LU-13508</del></a> mdc: chlg device could be used after free<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1e992e94eaf8ac9db2f6a7405cb9b8daf72ed742</p> <p>Landed for 2.14</p> Related LU-12506 Development Rank 1|i00zd3: Rank (Obsolete) 9223372036854775807 Severity