https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11280 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/299d644c-a6de-11e8-a5f2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/299d644c-a6de-11e8-a5f2-52540065bddc</a></p> <p>test_56w failed because MDS crashed with following stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2736.998005] BUG: unable to handle kernel paging request at ffff9ed8c35f3714 [ 2736.998890] IP: [&lt;ffffffffc10884e3&gt;] lod_qos_prep_create+0x5d3/0x17a0 [lod] [ 2736.999731] PGD 5442e067 PUD 0 [ 2737.000131] Oops: 0000 [#1] SMP [ 2737.000531] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core sunrpc dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd i2c_piix4 parport_pc joydev i2c_core pcspkr virtio_balloon parport ip_tables ext4 ata_generic mbcache pata_acpi jbd2 ata_piix virtio_blk libata 8139too crct10dif_pclmul crct10dif_common [ 2737.009634] crc32c_intel floppy serio_raw virtio_pci 8139cp virtio_ring virtio mii [ 2737.010477] CPU: 0 PID: 4463 Comm: mdt00_000 Kdump: loaded Tainted: G OE ------------ 3.10.0-862.9.1.el7_lustre.x86_64 #1 [ 2737.011730] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 2737.012337] task: ffff9ed89ec00000 ti: ffff9ed8a5aec000 task.ti: ffff9ed8a5aec000 [ 2737.013118] RIP: 0010:[&lt;ffffffffc10884e3&gt;] [&lt;ffffffffc10884e3&gt;] lod_qos_prep_create+0x5d3/0x17a0 [lod] [ 2737.014122] RSP: 0018:ffff9ed8a5aef5f0 EFLAGS: 00010293 [ 2737.014671] RAX: ffff9ed8b84bfdb0 RBX: 000000005899cae8 RCX: ffff9ed85899cad8 [ 2737.015404] RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff9ed89c1fd200 [ 2737.016147] RBP: ffff9ed8a5aef6e0 R08: ffff9ed89e2bc550 R09: 0000000000000000 [ 2737.016876] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000000 [ 2737.017620] R13: 0000000000000002 R14: ffff9ed8b7e00000 R15: ffff9ed8b97a1fc0 [ 2737.018378] FS: 0000000000000000(0000) GS:ffff9ed8bfc00000(0000) knlGS:0000000000000000 [ 2737.019224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2737.019832] CR2: ffff9ed8c35f3714 CR3: 0000000036372000 CR4: 00000000000606f0 [ 2737.020590] Call Trace: [ 2737.020908] [&lt;ffffffffc0d74d70&gt;] ? qsd_op_begin+0xb0/0x4d0 [lquota] [ 2737.021581] [&lt;ffffffffc10898c5&gt;] lod_prepare_create+0x215/0x2e0 [lod] [ 2737.022273] [&lt;ffffffffc107b7ee&gt;] lod_declare_striped_create+0x1ee/0x980 [lod] [ 2737.023039] [&lt;ffffffffc1089e8f&gt;] ? lod_sub_declare_create+0xdf/0x210 [lod] [ 2737.023764] [&lt;ffffffffc107fec4&gt;] lod_declare_create+0x204/0x590 [lod] [ 2737.024456] [&lt;ffffffffc106f412&gt;] ? lod_striping_from_default+0x492/0x5b0 [lod] [ 2737.025335] [&lt;ffffffffc094e9d9&gt;] ? lu_context_refill+0x19/0x50 [obdclass] [ 2737.026101] [&lt;ffffffffc10f2892&gt;] mdd_declare_create_object_internal+0xe2/0x2f0 [mdd] [ 2737.026909] [&lt;ffffffffc10e21c8&gt;] mdd_declare_create+0x48/0xc10 [mdd] [ 2737.027590] [&lt;ffffffffc10e65e9&gt;] mdd_create+0x929/0x13f0 [mdd] [ 2737.028284] [&lt;ffffffffc0f91e37&gt;] mdt_reint_open+0x2117/0x3160 [mdt] [ 2737.028973] [&lt;ffffffffc09634af&gt;] ? upcall_cache_get_entry+0x3df/0x8b0 [obdclass] [ 2737.029767] [&lt;ffffffffc0f85ce3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [ 2737.030409] [&lt;ffffffffc0f651d2&gt;] mdt_reint_internal+0x6b2/0xa80 [mdt] [ 2737.031100] [&lt;ffffffffc0f716c2&gt;] mdt_intent_open+0x82/0x350 [mdt] [ 2737.031759] [&lt;ffffffffc092d6f9&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [ 2737.032527] [&lt;ffffffffc0f6f768&gt;] mdt_intent_policy+0x2f8/0xd10 [mdt] [ 2737.033224] [&lt;ffffffffc0f71640&gt;] ? mdt_intent_fixup_resent+0x220/0x220 [mdt] [ 2737.034120] [&lt;ffffffffc0b39e9e&gt;] ldlm_lock_enqueue+0x34e/0xa50 [ptlrpc] [ 2737.034863] [&lt;ffffffffc07516ee&gt;] ? cfs_hash_add+0xbe/0x1a0 [libcfs] [ 2737.035592] [&lt;ffffffffc0b62483&gt;] ldlm_handle_enqueue0+0x903/0x1520 [ptlrpc] [ 2737.036373] [&lt;ffffffffc0b8a2d0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [ 2737.037224] [&lt;ffffffffc0be8932&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [ 2737.037901] [&lt;ffffffffc0bf127a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 2737.038639] [&lt;ffffffffc0748ee7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 2737.039364] [&lt;ffffffffc0b9440b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 2737.040188] [&lt;ffffffff972c52ab&gt;] ? __wake_up_common+0x5b/0x90 [ 2737.040823] [&lt;ffffffffc0b97c44&gt;] ptlrpc_main+0xb14/0x1fb0 [ptlrpc] [ 2737.041488] [&lt;ffffffff972c9e50&gt;] ? finish_task_switch+0x50/0x170 [ 2737.042152] [&lt;ffffffffc0b97130&gt;] ? ptlrpc_register_service+0xe90/0xe90 [ptlrpc] [ 2737.042933] [&lt;ffffffff972bb621&gt;] kthread+0xd1/0xe0 [ 2737.043459] [&lt;ffffffff972bb550&gt;] ? insert_kthread_work+0x40/0x40 [ 2737.044119] [&lt;ffffffff979205f7&gt;] ret_from_fork_nospec_begin+0x21/0x21 [ 2737.044804] [&lt;ffffffff972bb550&gt;] ? insert_kthread_work+0x40/0x40 [ 2737.045452] Code: ff 89 1c 90 8b 45 98 31 d2 0f b7 4f 24 83 c0 01 f7 f1 41 39 cd 89 55 98 7d 25 48 8b 4f 30 8b 45 98 8b 1c 81 49 8b 86 88 09 00 00 &lt;0f&gt; a3 58 08 19 c0 85 c0 0f 85 17 ff ff ff 41 bc ed ff ff ff f6 [ 2737.048776] RIP [&lt;ffffffffc10884e3&gt;] lod_qos_prep_create+0x5d3/0x17a0 [lod] [ 2737.049546] RSP &lt;ffff9ed8a5aef5f0&gt; [ 2737.049919] CR2: ffff9ed8c35f3714 </pre> </div></div> LU-11280

sanity: test_56w: unable to handle kernel paging request in lod_qos_prep_create

Bug Blocker Resolved Duplicate Zhenyu Xu Maloo Fri, 24 Aug 2018 07:05:25 +0000 Wed, 19 Dec 2018 21:11:55 +0000 Tue, 4 Sep 2018 09:53:33 +0000 Lustre 2.12.0 0 7 <p>Although there were time outs for this test in July 2018, we see this crash in review-ldiskfs since August 23, 2018.</p> <p>A few more failures since this ticket was opened:<br/> <a href="https://testing.whamcloud.com/test_sets/e7abc4e2-a918-11e8-80f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e7abc4e2-a918-11e8-80f7-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/e068ec00-aab7-11e8-bd05-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e068ec00-aab7-11e8-bd05-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/47659a3c-aad3-11e8-80f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/47659a3c-aad3-11e8-80f7-52540065bddc</a></p> <p>There are a few crashes that have the same stack trace, but different instruction pointer</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2825.966155] BUG: unable to handle kernel NULL pointer dereference at 0000000000000380 [ 2825.966986] IP: [&lt;ffffffffc1025976&gt;] lod_statfs_and_check+0x66/0x590 [lod] </pre> </div></div> <p><a href="https://testing.whamcloud.com/test_sets/18b3c6b0-a956-11e8-bd05-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/18b3c6b0-a956-11e8-bd05-52540065bddc</a></p> <p>Bobijam</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Hitting this fairly often in testing.</p> <p>About 11% of review-ldiskfs sessions are currently failing because of this.</p> <p>Tried but failed to reproduce it. The memory access fault seems locates in lod_alloc_ost_list()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ... /* * We've successfully declared (reserved) an object */ lod_qos_ost_in_use(env, stripe_count, ost_idx); stripe[stripe_count] = o; ost_indices[stripe_count] = ost_idx; <span class="code-comment">// memory access fault here it seems. </span> stripe_count++; } RETURN(rc); } </pre> </div></div> <p>I haven't found the code defect here yet, the ost_indices and stripe array has been allocated enough space, and stripe_count should not beyond the arrays boundary.</p> <p>It looks like patch <a href="https://review.whamcloud.com/32814" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32814</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11146" title="setstripe for specific osts are broken" class="issue-link" data-issue-key="LU-11146"><del>LU-11146</del></a> lustre: fix setstripe for specific osts upon dir</tt>" is responsible for this failure. That patch failed once with this stack on 2018-08-07 before it was landed, and no other tests failed with the same stack until 2018-08-23 after it was landed. </p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33105" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33105</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11280" title="sanity: test_56w: unable to handle kernel paging request in lod_qos_prep_create" class="issue-link" data-issue-key="LU-11280"><del>LU-11280</del></a> revert: fix setstripe for specific osts upon dir"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 799514054df0f2c71d98e1480e31d2ccfc3b6713</p> <p>I am wondering whether <a href="https://review.whamcloud.com/#/c/33069/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33069/</a> this might be cause and fix the problem.</p> <p>It deserve to run sanity_27H together with 56W to reproduce this BUG.</p> <p>I just retriggered <a href="https://review.whamcloud.com/#/c/33069/6" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33069/6</a> to check<br/> whether that patch fixed the problem.</p> <p>Because it looks strange 56w make the path to lod_alloc_ostlist() and<br/> what above patch tried to address might make it happen.</p> <p>I just retriggered <a href="https://review.whamcloud.com/#/c/33069/6" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33069/6</a> to check<br/> whether that patch fixed the problem.</p> <p>Because it looks strange 56w make the path to lod_alloc_ostlist() and<br/> what above patch tried to address might make it happen.</p> <p>yes, I can reproduce it with test only 27H then 56w after removing <a href="https://jira.whamcloud.com/browse/LU-11279" title="sanity test_65c: lverify failed" class="issue-link" data-issue-key="LU-11279"><del>LU-11279</del></a> patch, and not hit it with <a href="https://jira.whamcloud.com/browse/LU-11279" title="sanity test_65c: lverify failed" class="issue-link" data-issue-key="LU-11279"><del>LU-11279</del></a> patch in place. So I think and verified with my VM that <a href="https://jira.whamcloud.com/browse/LU-11279" title="sanity test_65c: lverify failed" class="issue-link" data-issue-key="LU-11279"><del>LU-11279</del></a> fixes this issue.</p> <p>Great news! Then let's close this issue out as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11279" title="sanity test_65c: lverify failed" class="issue-link" data-issue-key="LU-11279"><del>LU-11279</del></a></p> Duplicate LU-11279 Related LU-10279 LU-11146 LU-11279 Development Rank 1|i0017r: Rank (Obsolete) 9223372036854775807 Severity