https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12491 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/0f90891c-9a35-11e9-b26a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0f90891c-9a35-11e9-b26a-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[20823.002620] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 [20823.428725] general protection fault: 0000 [#1] SMP [20823.429775] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core sunrpc dm_mod ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables ext4 mbcache jbd2 virtio_blk ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw floppy ata_piix 8139too [20823.443544] libata virtio_pci virtio_ring virtio 8139cp mii [last unloaded: dm_flakey] [20823.444878] CPU: 1 PID: 19396 Comm: mdt_out00_003 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.21.3.el7_lustre.x86_64 #1 [20823.446885] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [20823.447816] task: ffff966f9f1d6180 ti: ffff966f89258000 task.ti: ffff966f89258000 [20823.449023] RIP: 0010:[&lt;ffffffffad380299&gt;] [&lt;ffffffffad380299&gt;] memcmp+0x9/0x50 [20823.450254] RSP: 0018:ffff966f8925bd98 EFLAGS: 00010202 [20823.451115] RAX: 00000000ffffffe0 RBX: 5a5a5a5a5a5a5a5a RCX: 0000000000000008 [20823.452265] RDX: 0000000000000008 RSI: ffff966f8925bdb0 RDI: 5a5a5a5a5a5a5a52 [20823.453414] RBP: ffff966f8925bd98 R08: ffffffffadd589e0 R09: 0000000000000000 [20823.454568] R10: ffff966fbfb80f20 R11: ffffffffffffffec R12: ffff966fb7845000 [20823.455711] R13: fffffffffffffff8 R14: 5a5a5a5a5a5a5a52 R15: 0000000000000000 [20823.456859] FS: 0000000000000000(0000) GS:ffff966fbfd00000(0000) knlGS:0000000000000000 [20823.458155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20823.459081] CR2: 00007fd5f82ab630 CR3: 000000005c520000 CR4: 00000000000606e0 [20823.460224] Call Trace: [20823.460790] [&lt;ffffffffc0a4eee7&gt;] lu_env_remove+0x147/0x380 [obdclass] [20823.462141] [&lt;ffffffffc0d3cb10&gt;] ptlrpc_main+0x5f0/0x1560 [ptlrpc] [20823.463174] [&lt;ffffffffad0d09f0&gt;] ? finish_task_switch+0x50/0x1c0 [20823.464199] [&lt;ffffffffc0d3c520&gt;] ? ptlrpc_register_service+0xfa0/0xfa0 [ptlrpc] [20823.465426] [&lt;ffffffffad0c1da1&gt;] kthread+0xd1/0xe0 [20823.466226] [&lt;ffffffffad0c1cd0&gt;] ? insert_kthread_work+0x40/0x40 [20823.467216] [&lt;ffffffffad775c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [20823.468271] [&lt;ffffffffad0c1cd0&gt;] ? insert_kthread_work+0x40/0x40 [20823.469259] Code: 66 90 55 31 c9 48 85 d2 48 89 f8 48 89 e5 74 0f 66 90 48 89 34 c8 48 83 c1 01 48 39 d1 75 f3 5d c3 90 55 48 85 d2 48 89 e5 74 3c &lt;0f&gt; b6 07 0f b6 0e 29 c8 75 27 48 83 ea 01 31 c9 eb 1a 0f 1f 44 [20823.474428] RIP [&lt;ffffffffad380299&gt;] memcmp+0x9/0x50 [20823.475292] RSP &lt;ffff966f8925bd98&gt; </pre> </div></div> LU-12491

large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50

Bug Major Resolved Fixed Alex Zhuravlev Maloo Sat, 29 Jun 2019 16:50:34 +0000 Fri, 26 Jul 2019 19:14:08 +0000 Fri, 12 Jul 2019 12:46:13 +0000 Lustre 2.13.0 Lustre 2.13.0 Lustre 2.12.3 0 7 <p>regression introduced by <br/> <a href="https://jira.whamcloud.com/browse/LU-12034" title="env allocation in ptlrpc_set_wait() causes slowdown on the client" class="issue-link" data-issue-key="LU-12034"><del>LU-12034</del></a> aka <a href="https://review.whamcloud.com/#/c/34566/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34566/</a>.</p> <p>I think we should don't make release with regressions a specially with panic.</p> <p>Alex</p> <p>Does this seem related to your recent change?</p> <p>Peter</p> <p>Peter, yes, it is, my bad, I've submitted a patch already.</p> <p>ah I see - <a href="https://review.whamcloud.com/#/c/35038/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35038/</a> . I wonder why JIRA did not have a comment about that...</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/35447" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35447</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: add comment for rcu handling in lu_env_remove<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4cb6fbab52bd7ce0b34098c8e7dddf4f0778ebd6</p> <p>Shaun, please clarify did you hit this with the latest or previous version of the patch?<br/> thanks for testing!</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35038/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35038/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: use RCU to release lu_env_item<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 87306c22e4b977356f4857d5f750447639d89c26</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35447/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35447/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: add comment for rcu handling in lu_env_remove<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 709fbe6ee54aa2e601237a6981db3d42a8a719cd</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35487" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35487</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: use RCU to release lu_env_item<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a0363369b2c25ef0146a40c13e7aeaafe2b060e3</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35488" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35488</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: add comment for rcu handling in lu_env_remove<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c2b43c331e747e97e235e63f4f619ff24284972f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35487/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35487/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: use RCU to release lu_env_item<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: cf2b060ccad73359151867ad419e36f746c9f241</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35488/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35488/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12491" title="large-scale: crash in lu_env_remove() RIP: memcmp+0x9/0x50" class="issue-link" data-issue-key="LU-12491"><del>LU-12491</del></a> obdclass: add comment for rcu handling in lu_env_remove<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: c579e6f6c28d0480bd4a815d9ad9134d3c913343</p> Duplicate LU-12497 Related LU-12034 Development Rank 1|i00ixz: Rank (Obsolete) 9223372036854775807 Severity