https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14434 Lustre <p>We’ve only seen this crash twice; only in parallel-scale-nfsv4 test compilebench:<br/> 2021-01-22: x86_64 clients - <a href="https://testing.whamcloud.com/test_sets/68ed07f9-eb1d-459c-b327-269bd996d449" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/68ed07f9-eb1d-459c-b327-269bd996d449</a><br/> 2021-02-11: ARM clients - <a href="https://testing.whamcloud.com/test_sets/6aae8467-c5e3-4547-aefa-04f220cf4042" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6aae8467-c5e3-4547-aefa-04f220cf4042</a></p> <p>Looking at the first failure above, we see in the kernel-crash</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[47446.646161] Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench ============================================== 19:11:27 (1611342687) [47447.194275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.parallel-scale-nfs\/d0.compilebench.1394887 -i 2 -r 2 --makej [47447.620295] Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.parallel-scale-nfs/d0.compilebench.1394887 -i 2 -r 2 --makej [48153.651976] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 [48153.653435] PGD 0 P4D 0 [48153.653864] Oops: 0000 [#1] SMP PTI [48153.654527] CPU: 0 PID: 1485996 Comm: qmt_reba_lustre Kdump: loaded Tainted: G OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [48153.656584] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [48153.657600] RIP: 0010:qmt_id_lock_cb+0x69/0x100 [lquota] [48153.658462] Code: 48 8b 53 20 8b 4a 0c 85 c9 74 74 89 c1 48 8b 42 18 83 78 10 02 75 0a 83 e1 01 b8 01 00 00 00 74 17 48 63 44 24 04 48 c1 e0 04 &lt;48&gt; 03 45 00 f6 40 08 0c 0f 95 c0 0f b6 c0 48 8b 4c 24 08 65 48 33 [48153.661475] RSP: 0018:ffffbf43c0c5bde8 EFLAGS: 00010246 [48153.662317] RAX: 0000000000000000 RBX: ffff9fbe4b55e000 RCX: 0000000000000000 [48153.663454] RDX: ffff9fbe71e8f7a0 RSI: 0000000000000000 RDI: ffff9fbe47c2e862 [48153.664587] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000004 [48153.665717] R10: 0000000000000010 R11: f000000000000000 R12: ffff9fbe4b55e000 [48153.666855] R13: ffff9fbe3133be60 R14: ffff9fbe4ebacb98 R15: ffff9fbe4ebacb40 [48153.667999] FS: 0000000000000000(0000) GS:ffff9fbe7fc00000(0000) knlGS:0000000000000000 [48153.669283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [48153.670208] CR2: 0000000000000000 CR3: 0000000059c0a002 CR4: 00000000000606f0 [48153.671348] Call Trace: [48153.671787] ? cfs_cdebug_show.part.2.constprop.22+0x20/0x20 [lquota] [48153.672831] qmt_glimpse_lock.isra.19+0x27e/0xfb0 [lquota] [48153.673726] qmt_reba_thread+0x5da/0x9b0 [lquota] [48153.674503] ? qmt_glimpse_lock.isra.19+0xfb0/0xfb0 [lquota] [48153.675454] kthread+0x112/0x130 [48153.676009] ? kthread_flush_work_fn+0x10/0x10 [48153.676745] ret_from_fork+0x35/0x40 [48153.677349] Modules linked in: nfsd nfs_acl lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul dm_mod ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 ip_tables ext4 mbcache jbd2 ata_generic 8139too ata_piix crc32c_intel libata serio_raw 8139cp mii virtio_blk [last unloaded: dm_flakey] [48153.688461] CR2: 0000000000000000 </pre> </div></div> LU-14434

parallel-scale-nfsv4 test compilebench crashes in qmt_id_lock_cb

Bug Minor Resolved Duplicate Sergey Cheremencev James Nunez quota Mon, 15 Feb 2021 22:05:29 +0000 Thu, 15 Jun 2023 14:55:57 +0000 Thu, 15 Jun 2023 14:55:57 +0000 Lustre 2.14.0 0 3 <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/b92d9d88-7878-447a-8d28-0f7e6e0a8c9b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/b92d9d88-7878-447a-8d28-0f7e6e0a8c9b</a></p> <p>This problem regularly appears during testing of <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49239" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49239</a></p> <p>I've added "+quota+trace" debug into replay-single.sh to gather more logs and move on.<br/> If it will not help and the problem disappears I am going to add a fix with assertion in qmt_revalidate_lqes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/quota/qmt_entry.c b/lustre/quota/qmt_entry.c index de3c891a1b..d776efa3ab 100644 --- a/lustre/quota/qmt_entry.c +++ b/lustre/quota/qmt_entry.c @@ -927,6 +927,7 @@ void qmt_revalidate_lqes(const struct lu_env *env,                 return;         }   +       LASSERT(lqe_gl-&gt;lqe_glbl_data);         mutex_lock(&amp;lqe_gl-&gt;lqe_glbl_data_lock);         if (lqe_gl-&gt;lqe_glbl_data)                 qmt_seed_glbe(env, lqe_gl-&gt;lqe_glbl_data); </pre> </div></div> <p>I guess the issue comes from this place.</p> <p><a href="https://review.whamcloud.com/c/fs/lustre-release/+/50748" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50748</a> since patchset 8 should help with current failure</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-16772" title="Protect lqe_glbl_data in qmt_site_recalc_cb with mutex" class="issue-link" data-issue-key="LU-16772"><del>LU-16772</del></a></p> Related LU-16772 Development Rank 1|i01mo7: Rank (Obsolete) 9223372036854775807 Severity