[LU-18657] Bad page state in sanity pcc test 21j

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Fix Version/s: Lustre 2.17.0
• Labels: janitor9x
• Severity: 3
• Assignee: WC Triage
• Reporter: Oleg Drokin

Description

Testing rhel 8.10 kernel sanity-pcc test 21j emit the following warning 100% of the time:

[ 1392.075674] Lustre: DEBUG MARKER: == sanity-pcc test 21i: HSM release increase layout gen, should invalidate PCC-RO cache ========================================================== 01:45:51 (1737269151)
[ 1392.775706] loop0: detected capacity change from 0 to 52428800
[ 1392.794194] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: usrquota,grpquota
[ 1400.966081] Lustre: DEBUG MARKER: == sanity-pcc test 21j: PCC-RO for encrypted file ======== 01:46:00 (1737269160)
[ 1401.684123] loop0: detected capacity change from 0 to 52428800
[ 1401.702825] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: usrquota,grpquota
[ 1402.356208] Lustre: Unmounted lustre-client
[ 1402.357105] Lustre: Skipped 1 previous similar message
[ 1402.571556] Lustre: Test dummy encryption mode enabled
[ 1402.579624] Lustre: Mounted lustre-client
[ 1402.580409] Lustre: Skipped 1 previous similar message
[ 1402.874696] llcrypt: AES-256-CTS-CBC using implementation "cts(cbc-aes-aesni)"
[ 1403.153064] llcrypt: AES-256-XTS using implementation "xts-aes-aesni"
[ 1403.718421] BUG: Bad page state in process lfs  pfn:1222d8
[ 1403.719741] page:ffffdd108488b600 refcount:0 mapcount:0 mapping:0000000000000000 index:0x1
[ 1403.721323] flags: 0x17ffffc000200c(referenced|uptodate|private_2|node=0|zone=2|lastcpupid=0x1fffff)
[ 1403.723651] raw: 0017ffffc000200c ffffdd1084b19688 ffffdd1084b7bc48 0000000000000000
[ 1403.725701] raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000
[ 1403.727433] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set
[ 1403.729136] Modules linked in: cts loop lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
[ 1403.739540] CPU: 3 PID: 66512 Comm: lfs Kdump: loaded Tainted: G        W  O      -------- -  - 4.18.0rh8.10-debug #7
[ 1403.741972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
[ 1403.743877] Call Trace:
[ 1403.744619]  ? dump_stack+0xbb/0x10e
[ 1403.745600]  ? bad_page.cold.105+0x8b/0xde
[ 1403.746422]  ? check_free_page_bad+0x2e/0x40
[ 1403.747408]  ? free_unref_page_prepare+0x4ad/0x4f0
[ 1403.748320]  ? free_unref_page_list+0x75/0x440
[ 1403.749332]  ? release_pages+0x271/0x8c0
[ 1403.750345]  ? __pagevec_release+0x23/0x50
[ 1403.751208]  ? truncate_inode_pages_range+0x452/0xeb0
[ 1403.752521]  ? snprintf+0x51/0x70
[ 1403.753192]  ? _raw_spin_unlock+0x3f/0x60
[ 1403.754289]  ? cfs_trace_unlock_tcd+0x2b/0xa0 [libcfs]
[ 1403.755373]  ? libcfs_debug_msg+0xcf5/0x1220 [libcfs]
[ 1403.756587]  ? __mutex_lock+0x15c/0xf80
[ 1403.757286]  ? memcg_slab_free_hook+0x1e0/0x2f0
[ 1403.758060]  ? truncate_inode_pages_final+0x5e/0x80
[ 1403.759152]  ? pcc_inode_put+0x168/0x240 [lustre]
[ 1403.760368]  ? pcc_file_release+0x1c7/0x4b0 [lustre]
[ 1403.761521]  ? ll_file_release+0xd0/0xe30 [lustre]
[ 1403.762563]  ? __fput+0x116/0x450
[ 1403.763181]  ? ____fput+0x12/0x20
[ 1403.763745]  ? task_work_run+0xc6/0x110
[ 1403.764615]  ? exit_to_usermode_loop+0x1dd/0x1f0
[ 1403.765658]  ? do_syscall_64+0x3e3/0x450
[ 1403.766469]  ? entry_SYSCALL_64_after_hwframe+0x49/0xae
[ 1403.767654] BUG: Bad page state in process lfs  pfn:12c65a
[ 1403.769216] page:ffffdd1084b19680 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0
[ 1403.770726] flags: 0x17ffffc000200c(referenced|uptodate|private_2|node=0|zone=2|lastcpupid=0x1fffff)
[ 1403.772716] raw: 0017ffffc000200c ffffb20982b17b60 ffffdd1084b7bc48 0000000000000000
[ 1403.774148] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[ 1403.775929] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set
[ 1403.777281] Modules linked in: cts loop lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
[ 1403.786979] CPU: 3 PID: 66512 Comm: lfs Kdump: loaded Tainted: G    B   W  O      -------- -  - 4.18.0rh8.10-debug #7
[ 1403.788996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
[ 1403.791035] Call Trace:
[ 1403.791570]  ? dump_stack+0xbb/0x10e
[ 1403.792349]  ? bad_page.cold.105+0x8b/0xde
[ 1403.793124]  ? check_free_page_bad+0x2e/0x40
[ 1403.793918]  ? free_unref_page_prepare+0x4ad/0x4f0
[ 1403.795394]  ? free_unref_page_list+0x75/0x440
[ 1403.796240]  ? release_pages+0x271/0x8c0
[ 1403.797103]  ? __pagevec_release+0x23/0x50
[ 1403.797981]  ? truncate_inode_pages_range+0x452/0xeb0
[ 1403.798859]  ? snprintf+0x51/0x70
[ 1403.799437]  ? _raw_spin_unlock+0x3f/0x60
[ 1403.800206]  ? cfs_trace_unlock_tcd+0x2b/0xa0 [libcfs]
[ 1403.801451]  ? libcfs_debug_msg+0xcf5/0x1220 [libcfs]
[ 1403.802655]  ? __mutex_lock+0x15c/0xf80
[ 1403.803363]  ? memcg_slab_free_hook+0x1e0/0x2f0
[ 1403.804173]  ? truncate_inode_pages_final+0x5e/0x80
[ 1403.805203]  ? pcc_inode_put+0x168/0x240 [lustre]
[ 1403.806162]  ? pcc_file_release+0x1c7/0x4b0 [lustre]
[ 1403.807522]  ? ll_file_release+0xd0/0xe30 [lustre]
[ 1403.808694]  ? __fput+0x116/0x450
[ 1403.809446]  ? ____fput+0x12/0x20
[ 1403.810049]  ? task_work_run+0xc6/0x110
[ 1403.810709]  ? exit_to_usermode_loop+0x1dd/0x1f0
[ 1403.811718]  ? do_syscall_64+0x3e3/0x450
[ 1403.812618]  ? entry_SYSCALL_64_after_hwframe+0x49/0xae
[ 1404.522887] BUG: Bad page state in process ldlm_bl_03  pfn:12d0e5
[ 1404.524157] page:ffffdd1084b43940 refcount:0 mapcount:0 mapping:0000000000000000 index:0x1
[ 1404.525616] flags: 0x17ffffc000200c(referenced|uptodate|private_2|node=0|zone=2|lastcpupid=0x1fffff)
[ 1404.527317] raw: 0017ffffc000200c ffffdd1084e4d688 ffffdd1084882f08 0000000000000000
[ 1404.528653] raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000
[ 1404.530037] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set
[ 1404.531206] Modules linked in: cts loop lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
[ 1404.539433] CPU: 0 PID: 52562 Comm: ldlm_bl_03 Kdump: loaded Tainted: G    B   W  O      -------- -  - 4.18.0rh8.10-debug #7
[ 1404.541592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
[ 1404.543127] Call Trace:
[ 1404.543610]  ? dump_stack+0xbb/0x10e
[ 1404.544269]  ? bad_page.cold.105+0x8b/0xde
[ 1404.545004]  ? check_free_page_bad+0x2e/0x40
[ 1404.545801]  ? free_unref_page_prepare+0x4ad/0x4f0
[ 1404.546700]  ? free_unref_page_list+0x75/0x440
[ 1404.547522]  ? release_pages+0x271/0x8c0
[ 1404.548231]  ? __pagevec_release+0x23/0x50
[ 1404.548975]  ? truncate_inode_pages_range+0x452/0xeb0
[ 1404.549865]  ? string_nocheck+0x77/0xa0
[ 1404.550557]  ? string+0x58/0x70
[ 1404.551111]  ? vsnprintf+0x30a/0x7f0
[ 1404.551761]  ? snprintf+0x51/0x70
[ 1404.552367]  ? pcc_layout_invalidate+0x5d/0x790 [lustre]
[ 1404.553479]  ? __mutex_lock+0x15c/0xf80
[ 1404.554177]  ? truncate_inode_pages_final+0x5e/0x80
[ 1404.555219]  ? pcc_inode_put+0x168/0x240 [lustre]
[ 1404.556148]  ? pcc_layout_invalidate+0x1b7/0x790 [lustre]
[ 1404.557232]  ? vvp_conf_set+0xf4/0x130 [lustre]
[ 1404.558236]  ? cl_conf_set+0x15b/0x1c0 [obdclass]
[ 1404.559226]  ? ll_layout_conf+0x1f7/0x590 [lustre]
[ 1404.560311]  ? ll_lock_cancel_bits+0x939/0xe70 [lustre]
[ 1404.561343]  ? ll_lock_cancel_bits+0x939/0xe70 [lustre]
[ 1404.562423]  ? ll_md_blocking_ast+0x383/0x660 [lustre]
[ 1404.563459]  ? do_raw_spin_unlock+0x75/0x190
[ 1404.564518]  ? ldlm_cancel_callback+0x9b/0x3d0 [ptlrpc]
[ 1404.565684]  ? ldlm_cli_cancel_local+0xd2/0x600 [ptlrpc]
[ 1404.566838]  ? ldlm_cli_cancel_list_local+0x162/0x440 [ptlrpc]
[ 1404.568038]  ? do_raw_spin_unlock+0x75/0x190
[ 1404.568837]  ? ldlm_bl_thread_main+0xb13/0xd00 [ptlrpc]
[ 1404.569957]  ? woken_wake_function+0x30/0x30
[ 1404.570783]  ? ldlm_handle_bl_callback+0x500/0x500 [ptlrpc]
[ 1404.571955]  ? kthread+0x1d7/0x210
[ 1404.572588]  ? set_kthread_struct+0x70/0x70
[ 1404.573402]  ? ret_from_fork+0x1f/0x30
[ 1404.574197] BUG: Bad page state in process ldlm_bl_03  pfn:13935a
[ 1404.575361] page:ffffdd1084e4d680 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0
[ 1404.576889] flags: 0x17ffffc000200c(referenced|uptodate|private_2|node=0|zone=2|lastcpupid=0x1fffff)
[ 1404.578699] raw: 0017ffffc000200c ffffb20982b0f938 ffffdd1084882f08 0000000000000000
[ 1404.580178] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[ 1404.581634] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set
[ 1404.582890] Modules linked in: cts loop lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
[ 1404.591416] CPU: 0 PID: 52562 Comm: ldlm_bl_03 Kdump: loaded Tainted: G    B   W  O      -------- -  - 4.18.0rh8.10-debug #7
[ 1404.593511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
[ 1404.595178] Call Trace:
[ 1404.595631]  ? dump_stack+0xbb/0x10e
[ 1404.596293]  ? bad_page.cold.105+0x8b/0xde
[ 1404.597024]  ? check_free_page_bad+0x2e/0x40
[ 1404.597839]  ? free_unref_page_prepare+0x4ad/0x4f0
[ 1404.598782]  ? free_unref_page_list+0x75/0x440
[ 1404.599647]  ? release_pages+0x271/0x8c0
[ 1404.600360]  ? __pagevec_release+0x23/0x50
[ 1404.601098]  ? truncate_inode_pages_range+0x452/0xeb0
[ 1404.602086]  ? string_nocheck+0x77/0xa0
[ 1404.602821]  ? string+0x58/0x70
[ 1404.603398]  ? vsnprintf+0x30a/0x7f0
[ 1404.604050]  ? snprintf+0x51/0x70
[ 1404.604841]  ? pcc_layout_invalidate+0x5d/0x790 [lustre]
[ 1404.605914]  ? __mutex_lock+0x15c/0xf80
[ 1404.606687]  ? truncate_inode_pages_final+0x5e/0x80
[ 1404.607647]  ? pcc_inode_put+0x168/0x240 [lustre]
[ 1404.608563]  ? pcc_layout_invalidate+0x1b7/0x790 [lustre]
[ 1404.609620]  ? vvp_conf_set+0xf4/0x130 [lustre]
[ 1404.610666]  ? cl_conf_set+0x15b/0x1c0 [obdclass]
[ 1404.611668]  ? ll_layout_conf+0x1f7/0x590 [lustre]
[ 1404.612596]  ? ll_lock_cancel_bits+0x939/0xe70 [lustre]
[ 1404.613704]  ? ll_lock_cancel_bits+0x939/0xe70 [lustre]
[ 1404.614832]  ? ll_md_blocking_ast+0x383/0x660 [lustre]
[ 1404.615843]  ? do_raw_spin_unlock+0x75/0x190
[ 1404.616618]  ? ldlm_cancel_callback+0x9b/0x3d0 [ptlrpc]
[ 1404.617763]  ? ldlm_cli_cancel_local+0xd2/0x600 [ptlrpc]
[ 1404.618994]  ? ldlm_cli_cancel_list_local+0x162/0x440 [ptlrpc]
[ 1404.620225]  ? do_raw_spin_unlock+0x75/0x190
[ 1404.620992]  ? ldlm_bl_thread_main+0xb13/0xd00 [ptlrpc]
[ 1404.622172]  ? woken_wake_function+0x30/0x30
[ 1404.622951]  ? ldlm_handle_bl_callback+0x500/0x500 [ptlrpc]
[ 1404.624104]  ? kthread+0x1d7/0x210
[ 1404.624713]  ? set_kthread_struct+0x70/0x70
[ 1404.625491]  ? ret_from_fork+0x1f/0x30

Example run: http://testing.linuxhacker.ru/lustre-reports/48666/testresults/sanity-pcc-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64/

 

Issue Links

• is related to: LU-16807 Resolve newer debug kernel warnings