https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14306 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/c31eb44e-b299-487e-99da-794b128f0cc5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c31eb44e-b299-487e-99da-794b128f0cc5</a></p> <p>Even if it does not fail, test_52 shows the following error in the console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 694.000072] BUG: Bad rss-counter state mm:0000000049598c66 idx:1 val:2 [ 694.001372] BUG: Bad rss-counter state mm:0000000049598c66 idx:3 val:-2 [ 694.009960] BUG: Bad rss-counter state mm:0000000049598c66 idx:1 val:2 [ 694.011094] BUG: Bad rss-counter state mm:0000000049598c66 idx:3 val:-2 [ 694.092732] BUG: Bad rss-counter state mm:00000000e37c077b idx:1 val:2 [ 694.094024] BUG: Bad rss-counter state mm:00000000e37c077b idx:3 val:-2 [ 694.124577] BUG: Bad rss-counter state mm:0000000049598c66 idx:1 val:2 [ 694.125755] BUG: Bad rss-counter state mm:0000000049598c66 idx:3 val:-2 [ 694.367283] BUG: Bad rss-counter state mm:0000000049598c66 idx:1 val:3 [ 694.368441] BUG: Bad rss-counter state mm:0000000049598c66 idx:3 val:-3 [ 694.376602] BUG: Bad rss-counter state mm:0000000084701c1b idx:1 val:3 [ 694.377761] BUG: Bad rss-counter state mm:0000000084701c1b idx:3 val:-3 </pre> </div></div> <p>Also, from time to time, some pages are dumped, because of a nonzero mapcount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 700.614727] BUG: Bad page state in process lfs pfn:3ca01 [ 700.615774] page:ffffd87ac0f28040 refcount:0 mapcount:-1 mapping:0000000000000000 index:0x0 [ 700.617194] flags: 0xfffffc0000000() [ 700.617850] raw: 000fffffc0000000 0000000000000000 ffffffffc0f20903 0000000000000000 [ 700.622001] raw: 0000000000000000 0000000000000000 00000000fffffffe 0000000000000000 [ 700.623329] page dumped because: nonzero mapcount [ 700.624155] Modules linked in: lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp rpcrdma ib_iser ib_ipoib libiscsi scsi_transport_iscsi rdma_ucm ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_uverbs ib_core sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_piix4 virtio_balloon joydev pcspkr ip_tables ext4 mbcache jbd2 mlx4_en ata_generic ata_piix 8139too mlx4_core libata crc32c_intel serio_raw 8139cp virtio_blk mii [ 700.633949] CPU: 1 PID: 45813 Comm: lfs Kdump: loaded Tainted: G OE --------- - - 4.18.0-240.1.1.el8_3.x86_64 #1 [ 700.635821] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 700.636800] Call Trace: [ 700.637382] dump_stack+0x5c/0x80 [ 700.638015] bad_page.cold.115+0xba/0xbf [ 700.638726] __free_pages_ok+0x2c4/0x2d0 [ 700.639436] release_pages+0x305/0x430 [ 700.640117] ? cpumask_any_but+0x20/0x40 [ 700.640819] tlb_flush_mmu_free+0x3d/0x60 [ 700.641539] arch_tlb_finish_mmu+0x89/0x130 [ 700.642277] tlb_finish_mmu+0x1f/0x30 [ 700.642931] unmap_region+0xdd/0x110 [ 700.643585] ? __vma_rb_erase+0x127/0x250 [ 700.644294] do_munmap+0x256/0x440 [ 700.644913] vm_munmap+0x5f/0xa0 [ 700.645499] __x64_sys_munmap+0x22/0x30 [ 700.646208] do_syscall_64+0x5b/0x1a0 [ 700.646887] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 700.647778] RIP: 0033:0x7f4035528a0b [ 700.648418] Code: ff ff 0f 1f 44 00 00 48 8b 15 79 54 2c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff e9 6f ff ff ff f3 0f 1e fa b8 0b 00 00 00 0f 05 &lt;48&gt; 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 54 2c 00 f7 d8 64 89 01 48 [ 700.651441] RSP: 002b:00007ffc550ab5f8 EFLAGS: 00000202 ORIG_RAX: 000000000000000b [ 700.652712] RAX: ffffffffffffffda RBX: 00007ffc550ab7e0 RCX: 00007f4035528a0b [ 700.653907] RDX: 0000000000000000 RSI: 0000000000402000 RDI: 00007f4034bfe000 [ 700.655109] RBP: 0000000000000002 R08: 0000000000401010 R09: 00007ffc550ab4c0 [ 700.656330] R10: 000000000000000a R11: 0000000000000202 R12: 0000000000000002 [ 700.657530] R13: 00007ffc550ab6d0 R14: 00007f4034bff000 R15: 00007ffc550ab7e0 [ 700.658764] Disabling lock debugging due to kernel taint </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-sec test_52 - </p> LU-14306

sanity-sec test_52: BUG: Bad rss-counter state

Bug Minor Resolved Fixed Sebastien Buisson Maloo Thu, 7 Jan 2021 10:18:29 +0000 Mon, 25 Jan 2021 19:35:28 +0000 Mon, 25 Jan 2021 19:35:28 +0000 Lustre 2.14.0 0 6 <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41159" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41159</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> tests: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 91b117e0916969466137e43c0d1bf57fdca6d681</p> <p>It seems possible that this problem is caused by the use of "fake" pages for handling O_DIRECT writes from <tt>lfs mirror resync</tt>? Is it possible that the pages are not being reset properly after use, or they are not being initialized correctly before use?</p> <p>Indeed, that is most likely because we twist the Lustre pages used for RPCs so that they have a proper mapping and index,<br/> suitable for encryption/decryption. This is done in order to be able to support O_DIRECT for encrypted files, which is the IO mode used when syncing mirrors.</p> <p>These messages start in <tt>test_44</tt> which appears to be the first subtest to invoke direct IO.</p> <p>You are right, I would better try to get traces from sanity-sec test_44 instead of test_52, because test_44 explicitly does O_DIRECT IOs whereas test_52 indirectly triggers that via calls to <tt>lfs mirror</tt>.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41164" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41164</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> tests: attempt 1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c10dae7f3c5909ff4a3b3e47d3f0561cf68ea161</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41165" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41165</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> tests: attempt 2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 509d599f967095ed7969b037a93f747c7b975c38</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41199" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41199</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> sec: get rid of bad rss-counter state messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9d94fafbadd6fa2e1f0ce554d911113fb91cb3c9</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41205" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41205</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> sec: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0b249bdf4981e915083c9f116649f392582cdc30</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41208" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41208</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> sec: debug patch 2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cb8a25627b68d5cfc2d962e61f9aef815ab8d97b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41199/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41199/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14306" title="sanity-sec test_52: BUG: Bad rss-counter state" class="issue-link" data-issue-key="LU-14306"><del>LU-14306</del></a> sec: get rid of bad rss-counter state messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a71e0dd7f7aa445dc06a99c0a1ef34383cafa2e2</p> <p>Landed for 2.14</p> Related LU-14045 Development Rank 1|i01it3: Rank (Obsolete) 9223372036854775807 Severity