https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9433 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/cb12c60c-613a-44b3-bfef-03c0651d2607" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/cb12c60c-613a-44b3-bfef-03c0651d2607</a></p> <p>Note: This was also seen in v2.8: <a href="https://jira.whamcloud.com/browse/LU-8279" title="sanity-scrub test_4b: @@@@@@ FAIL: Error in dmesg detected" class="issue-link" data-issue-key="LU-8279"><del>LU-8279</del></a></p> <p>This sanity-scrub subtest failure in test_6 was followed by the same failure in the next ~375 subtests (in sanity-scrub, sanity-benchmark, sanity-lfsck, sanityn, and sanity-hsm.</p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-48vm1.onyx.hpdd.intel.com,onyx-48vm2,onyx-48vm3,onyx-48vm7,onyx-48vm8 dmesg Kernel error detected: [11155.947772] VFS: Busy inodes after unmount of dm-1. Self-destruct in 5 seconds. Have a nice day... sanity-scrub test_6: @@@@@@ FAIL: Error in dmesg detected Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4931:error() = /usr/lib64/lustre/tests/test-framework.sh:5212:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5246:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5093:run_test() = /usr/lib64/lustre/tests/sanity-scrub.sh:773:main() </pre> </div></div> <p>Note: The "VFS: Busy inodes after unmount" message is also present in the MDS2/MDS4 console log.</p> onyx-48, full, DNE/ldiskfs<br/> &nbsp;&nbsp;EL7, master branch, v2.9.56.11, b3565 LU-9433

sanity-scrub test_6: Error in dmesg detected

Bug Critical Resolved Fixed nasf James Casper Tue, 2 May 2017 20:17:03 +0000 Sat, 10 Jun 2017 12:43:07 +0000 Sat, 10 Jun 2017 12:43:07 +0000 Lustre 2.10.0 Lustre 2.10.0 0 9 <p>More failure instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4215ab26-3090-11e7-8847-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4215ab26-3090-11e7-8847-5254006e85c2</a></p> <p>Wei Liu (wei3.liu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26967" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26967</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9433" title="sanity-scrub test_6: Error in dmesg detected" class="issue-link" data-issue-key="LU-9433"><del>LU-9433</del></a> test: rerun sanity-scrub to reproduce it<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1889b2073c087be7328044cfa309996721717ff8</p> <p>After adding umount/remount between each subtest, cannot reproduce the problem, any suggestion?</p> <p>Maloo report<br/> <a href="https://testing.hpdd.intel.com/test_sessions/74dc4308-1a49-494d-9472-c75d12430ca3" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/74dc4308-1a49-494d-9472-c75d12430ca3</a></p> <p>FYI: In tag 56, this was only seen in the following config: DNE/ldiskfs. In tag 57, sanity-scrub passed for DNE/ldiskfs. </p> <p>Thank you Jim. The debug patch I pushed is using DNE/ldiskfs config. Please let me know if you see the bug in other config or tests. Thanks.</p> <p>We had a sanity-scrub failure last night at <a href="https://testing.hpdd.intel.com/test_sets/999eb6ae-3607-11e7-8847-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/999eb6ae-3607-11e7-8847-5254006e85c2</a>; sanity-scrub test 9 timed out. When we run the same test suite multiple times in one patch (for example when we specify testlist=sanity-scrub,sanity-scrub), the logs get combined; see ATM-13. Thus, all the logs for all the sanity-scrub runs are combined and located at <a href="https://testing.hpdd.intel.com/test_sets/9a81d60a-3607-11e7-8847-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9a81d60a-3607-11e7-8847-5254006e85c2</a>. </p> <p>Looking at test 9 MDS1/MDS3 console logs, about half way through the log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:58:47:[15886.272894] Lustre: DEBUG MARKER: onyx-61vm3.onyx.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 15:58:47:[15887.708303] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 15:58:47:[15887.978941] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 16:58:33:********** Timeout by autotest system **********18:10:53:[ 2558.290504] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control </pre> </div></div> <p>Looking at the MDS2/MDS4 console log, if you search, you will find</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:59:31:[15925.512212] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0003.lfsck_speed_limit 300 15:59:31:[15936.070002] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [OI_scrub:28076] 15:59:31:[15936.070002] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr virtio_balloon i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk crct10dif_pclmul crct10dif_common cirrus crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops 8139too ttm serio_raw virtio_pci virtio_ring virtio ata_piix drm libata i2c_core 8139cp mii floppy 15:59:31:[15936.070002] CPU: 1 PID: 28076 Comm: OI_scrub Tainted: G OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 15:59:31:[15936.070002] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 15:59:31:[15936.070002] task: ffff880063478fb0 ti: ffff88007a8cc000 task.ti: ffff88007a8cc000 15:59:31:[15936.070002] RIP: 0010:[&lt;ffffffffa0d24a09&gt;] [&lt;ffffffffa0d24a09&gt;] osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] 15:59:31:[15936.070002] RSP: 0018:ffff88007a8cfc78 EFLAGS: 00000202 15:59:31:[15936.070002] RAX: 0000000000000000 RBX: ffff88007a8cfc37 RCX: 00000000555939a4 15:59:31:[15936.070002] RDX: ffff88007a8cfd78 RSI: ffff88005f5ce000 RDI: ffff880061399000 15:59:31:[15936.070002] RBP: ffff88007a8cfd08 R08: ffff88007a8cfd57 R09: 0000000000000004 15:59:31:[15936.070002] R10: ffff88007fd19a80 R11: ffffea0001e90800 R12: ffffffffa0d214d1 15:59:31:[15936.070002] R13: ffff88007a8cfc68 R14: ffff88005f5cf110 R15: ffff88007a420a00 15:59:31:[15936.070002] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000 15:59:31:[15936.070002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 15:59:31:[15936.070002] CR2: 00007f94144fa000 CR3: 00000000019be000 CR4: 00000000000406e0 15:59:31:[15936.070002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 15:59:31:[15936.070002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 15:59:31:[15936.070002] Stack: 15:59:31:[15936.070002] 0000000000000004 ffff88007a8cfd57 ffff88007a8cfdf0 0000000000000000 15:59:31:[15936.070002] ffff88007a8cfd57 ffff88007fd19a80 0000000000000004 ffff88007a8cfd57 15:59:31:[15936.070002] ffff880068c13800 ffff880068c13800 0000000000005404 ffff8800550dc000 15:59:31:[15936.070002] Call Trace: 15:59:31:[15936.070002] [&lt;ffffffffa0d261e9&gt;] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d24990&gt;] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d20a20&gt;] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d27370&gt;] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffff810c54c0&gt;] ? wake_up_state+0x20/0x20 15:59:31:[15936.070002] [&lt;ffffffffa0d26a10&gt;] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0 15:59:31:[15936.070002] [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 15:59:31:[15936.070002] [&lt;ffffffff81697318&gt;] ret_from_fork+0x58/0x90 15:59:31:[15936.070002] [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 15:59:31:[15936.070002] Code: 07 74 4f 41 83 f9 02 74 78 48 89 ca 44 89 c9 e8 0e c6 ff ff 85 c0 41 89 c1 0f 84 a3 01 00 00 41 80 a7 74 14 00 00 fe 48 8b 4d d0 &lt;65&gt; 48 33 0c 25 28 00 00 00 0f 85 b1 03 00 00 48 83 c4 68 5b 41 15:59:31:[15936.070002] Kernel panic - not syncing: softlockup: hung tasks 15:59:31:[15936.070002] CPU: 1 PID: 28076 Comm: OI_scrub Tainted: G OEL ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 15:59:31:[15936.070002] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 15:59:31:[15936.070002] ffffffff818da70d 00000000555939a4 ffff88007fd03e18 ffffffff81686d1f 15:59:31:[15936.070002] ffff88007fd03e98 ffffffff8168014a 0000000000000008 ffff88007fd03ea8 15:59:31:[15936.070002] ffff88007fd03e48 00000000555939a4 ffff88007fd03e67 0000000000000000 15:59:31:[15936.070002] Call Trace: 15:59:31:[15936.070002] &lt;IRQ&gt; [&lt;ffffffff81686d1f&gt;] dump_stack+0x19/0x1b 15:59:31:[15936.070002] [&lt;ffffffff8168014a&gt;] panic+0xe3/0x1f2 15:59:31:[15936.070002] [&lt;ffffffff8112f3bc&gt;] watchdog_timer_fn+0x20c/0x220 15:59:31:[15936.070002] [&lt;ffffffff8112f1b0&gt;] ? watchdog+0x50/0x50 15:59:31:[15936.070002] [&lt;ffffffff810b4d72&gt;] __hrtimer_run_queues+0xd2/0x260 15:59:31:[15936.070002] [&lt;ffffffff810b5310&gt;] hrtimer_interrupt+0xb0/0x1e0 15:59:31:[15936.070002] [&lt;ffffffff81698e5c&gt;] ? call_softirq+0x1c/0x30 15:59:31:[15936.070002] [&lt;ffffffff81050fd7&gt;] local_apic_timer_interrupt+0x37/0x60 15:59:31:[15936.070002] [&lt;ffffffff81699acf&gt;] smp_apic_timer_interrupt+0x3f/0x60 15:59:31:[15936.070002] [&lt;ffffffff8169801d&gt;] apic_timer_interrupt+0x6d/0x80 15:59:31:[15936.070002] &lt;EOI&gt; [&lt;ffffffffa0d24a09&gt;] ? osd_scrub_exec+0x79/0x5b0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d261e9&gt;] osd_inode_iteration+0x499/0xcc0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d24990&gt;] ? osd_ios_ROOT_scan+0x300/0x300 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d20a20&gt;] ? osd_preload_next+0xb0/0xb0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffffa0d27370&gt;] osd_scrub_main+0x960/0xf30 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffff810c54c0&gt;] ? wake_up_state+0x20/0x20 15:59:31:[15936.070002] [&lt;ffffffffa0d26a10&gt;] ? osd_inode_iteration+0xcc0/0xcc0 [osd_ldiskfs] 15:59:31:[15936.070002] [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0 15:59:31:[15936.070002] [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 15:59:31:[15936.070002] [&lt;ffffffff81697318&gt;] ret_from_fork+0x58/0x90 15:59:31:[15936.070002] [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>The stack trace is similar to <a href="https://jira.whamcloud.com/browse/LU-9488" title="soft lockup in osd_inode_iteration()" class="issue-link" data-issue-key="LU-9488"><del>LU-9488</del></a>.</p> <p>Fan Yong</p> <p>Can you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27212" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27212</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9433" title="sanity-scrub test_6: Error in dmesg detected" class="issue-link" data-issue-key="LU-9433"><del>LU-9433</del></a> osd-ldiskfs: fix inode reference leak<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d3204613fcc23ef97a632425d4ff49ce399b40ee</p> <p>The patch <a href="https://review.whamcloud.com/27212" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27212</a> is for fixing inode reference leak.<br/> As for the OI scrub soft lookup, it is another failure instance of <a href="https://jira.whamcloud.com/browse/LU-9488" title="soft lockup in osd_inode_iteration()" class="issue-link" data-issue-key="LU-9488"><del>LU-9488</del></a>, I will make another patch to fix it.</p> <p>The <a href="https://review.whamcloud.com/27228" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27228</a> is used to fix the OI scrub soft lockup issue.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27212/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27212/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9433" title="sanity-scrub test_6: Error in dmesg detected" class="issue-link" data-issue-key="LU-9433"><del>LU-9433</del></a> osd-ldiskfs: fix inode reference leak<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1cdb212683824ff24f8366c4e32efb559c46aee3</p> <p>Landed for 2.10</p> Related LU-8279 LU-5729 Development Rank 1|hzzbpb: Rank (Obsolete) 9223372036854775807 Severity