https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12832 Lustre <p>many clients got soft lockup in ldlm_bl_xx at read for a single shared strided file that is same workload of io500 hard. at the end, ior failed becouse of clients were evicted.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkdir /scratch0/io500.out/ior_hard -p # lfs setstripe -C 240 -S 16M /scratch0/io500.out/ior_hard # salloc --nodes=10 --ntasks-per-node=24 mpirun --allow-run-as-root /scratch0/ior -w -s 132000 -a POSIX -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /scratch0/io500.out/ior_hard/IOR_file -O stoneWallingStatusFile=/scratch0/io500.out/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 -W </pre> </div></div> <p>one of server log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 6 21:27:46 es400nv-vm1 kernel: LustreError: 26369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.0.13.151@o2ib10 ns: filter-ai400-OST0000_UUID lock: ffff995029f90480/0xe6d707f7408ef34e lrc: 3/0,0 mode: PW/PW res: [0x19:0x0:0x0].0x0 rrc: 30741 type: EXT [5842382848-&gt;5842853887] (req 5842382848-&gt;5842431999) flags: 0x60000400000020 nid: 10.0.13.151@o2ib10 remote: 0xeff507f1ffb50fc0 expref: 158923 pid: 27145 timeout: 11179 lvb_type: 0 Oct 6 21:27:46 es400nv-vm1 kernel: Lustre: ai400-OST0000: Connection restored to 85106299-51f6-4 (at 10.0.13.154@o2ib10) Oct 6 21:27:46 es400nv-vm1 kernel: Lustre: Skipped 127 previous similar messages Oct 6 21:27:49 es400nv-vm1 kernel: LustreError: 26369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.0.13.152@o2ib10 ns: filter-ai400-OST0001_UUID lock: ffff9950aee0d8c0/0xe6d707f740dbdd52 lrc: 3/0,0 mode: PW/PW res: [0x8:0x0:0x0].0x0 rrc: 26996 type: EXT [5140467712-&gt;5140746239] (req 5140467712-&gt;5140516863) flags: 0x60000400020020 nid: 10.0.13.152@o2ib10 remote: 0xc5564b8fed3e7bd7 expref: 166664 pid: 26904 timeout: 11182 lvb_type: 0 Oct 6 21:27:49 es400nv-vm1 kernel: LustreError: 26369:0:(ldlm_lockd.c:261:expired_lock_main()) Skipped 2 previous similar messages Oct 6 21:27:53 es400nv-vm1 kernel: LustreError: 26986:0:(ldlm_lockd.c:2437:ldlm_cancel_handler()) ldlm_cancel from 10.0.13.152@o2ib10 arrived at 1570364871 with bad export cookie 16633772507216549573 Oct 6 21:27:59 es400nv-vm1 kernel: LustreError: 26369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.0.13.159@o2ib10 ns: filter-ai400-OST0000_UUID lock: ffff9950587b6400/0xe6d707f7402a10dc lrc: 3/0,0 mode: PW/PW res: [0x15:0x0:0x0].0x0 rrc: 18417 type: EXT [6164672512-&gt;6164860927] (req 6164672512-&gt;6164721663) flags: 0x60000400020020 nid: 10.0.13.159@o2ib10 remote: 0xd7d9cb14933b97f6 expref: 153927 pid: 26985 timeout: 11192 lvb_type: 0</pre> </div></div> <p>one of client log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 6 21:27:54 c05 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [ldlm_bl_01:151183] Oct 6 21:27:54 c05 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) ip6table_nat ip6_tables xt_statistic xt_physdev xt_NFLOG nfnetlink_log veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ip_set_hash_ip xt_set ip_set xt_nat ipt_REJECT nf_reject_ipv4 xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) Oct 6 21:27:54 c05 kernel: mlx5_core(OE) mlxfw(OE) mlx4_en(OE) sunrpc iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg i2c_i801 mei_me lpc_ich ioatdma mei ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad acpi_power_meter binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci igb crct10dif_pclmul crct10dif_common libata crc32c_intel mlx4_core(OE) ptp pps_core devlink dca i2c_algo_bit drm_panel_orientation_quirks mlx_compat(OE) [last unloaded: libcfs] Oct 6 21:27:54 c05 kernel: CPU: 9 PID: 151183 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-957.27.2.el7.x86_64 #1 Oct 6 21:27:54 c05 kernel: Hardware name: Intel Corporation S2600KPR/S2600KPR, BIOS SE5C610.86B.01.01.0027.071020182329 07/10/2018 Oct 6 21:27:54 c05 kernel: task: ffff8eb4c1eb4100 ti: ffff8eb64c178000 task.ti: ffff8eb64c178000 Oct 6 21:27:54 c05 kernel: RIP: 0010:[&lt;ffffffff89d13536&gt;] [&lt;ffffffff89d13536&gt;] native_queued_spin_lock_slowpath+0x126/0x200 Oct 6 21:27:54 c05 kernel: RSP: 0018:ffff8eb64c17bc70 EFLAGS: 00000246 Oct 6 21:27:54 c05 kernel: RAX: 0000000000000000 RBX: ffff8ec614df8280 RCX: 0000000000490000 Oct 6 21:27:54 c05 kernel: RDX: ffff8eb67e79b780 RSI: 0000000000310001 RDI: ffff8ec614de0058 Oct 6 21:27:54 c05 kernel: RBP: ffff8eb64c17bc70 R08: ffff8eb67e85b780 R09: 0000000000000000 Oct 6 21:27:54 c05 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8eb64c17bc38 Oct 6 21:27:54 c05 kernel: R13: ffff8eb1aa758000 R14: ffff8eb2af3e46c8 R15: ffff8eb1aa758098 Oct 6 21:27:54 c05 kernel: FS: 0000000000000000(0000) GS:ffff8eb67e840000(0000) knlGS:0000000000000000 Oct 6 21:27:54 c05 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 6 21:27:54 c05 kernel: CR2: 00000000029cb000 CR3: 0000000f51610000 CR4: 00000000003607e0 Oct 6 21:27:54 c05 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 6 21:27:54 c05 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Oct 6 21:27:54 c05 kernel: Call Trace: Oct 6 21:27:54 c05 kernel: [&lt;ffffffff8a35e2cb&gt;] queued_spin_lock_slowpath+0xb/0xf Oct 6 21:27:54 c05 kernel: [&lt;ffffffff8a36c7a0&gt;] _raw_spin_lock+0x20/0x30 Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0c3b32a&gt;] cl_object_attr_lock+0x1a/0x20 [obdclass] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0fea746&gt;] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0e3e54a&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0c14979&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0e49460&gt;] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0e4d97a&gt;] ldlm_cli_cancel_list_local+0xea/0x260 [ptlrpc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0e54370&gt;] ldlm_bl_thread_main+0x5a0/0xa40 [ptlrpc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffff89cd7c40&gt;] ? wake_up_state+0x20/0x20 Oct 6 21:27:54 c05 kernel: [&lt;ffffffffc0e53dd0&gt;] ? ldlm_handle_bl_callback+0x4e0/0x4e0 [ptlrpc] Oct 6 21:27:54 c05 kernel: [&lt;ffffffff89cc2e81&gt;] kthread+0xd1/0xe0 Oct 6 21:27:54 c05 kernel: [&lt;ffffffff89cc2db0&gt;] ? insert_kthread_work+0x40/0x40 Oct 6 21:27:54 c05 kernel: [&lt;ffffffff8a376c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 Oct 6 21:27:54 c05 kernel: [&lt;ffffffff89cc2db0&gt;] ? insert_kthread_work+0x40/0x40 Oct 6 21:27:54 c05 kernel: Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 bf 94 8a 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 &lt;85&gt; c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 Oct 6 21:27:56 c05 kernel: LustreError: 152233:0:(ldlm_resource.c:1147:ldlm_resource_complain()) ai400-OST0006-osc-ffff8eafdda3d800: namespace resource [0x10:0x0:0x0].0x0 (ffff8ebf4ddcc840) refcount nonzero (1404) after lock cleanup; forcing cleanup. Oct 6 21:27:56 c05 kernel: Lustre: ai400-OST0006-osc-ffff8eafdda3d800: Connection restored to 10.0.10.178@o2ib10 (at 10.0.10.178@o2ib10) Oct 6 21:27:58 c05 kernel: LustreError: 152229:0:(ldlm_resource.c:1147:ldlm_resource_complain()) ai400-OST0002-osc-ffff8eafdda3d800: namespace resource [0x10:0x0:0x0].0x0 (ffff8ec4263355c0) refcount nonzero (1729) after lock cleanup; forcing cleanup. Oct 6 21:27:58 c05 kernel: LustreError: 152229:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 59 previous similar messages Oct 6 21:27:58 c05 kernel: Lustre: ai400-OST0002-osc-ffff8eafdda3d800: Connection restored to 10.0.10.176@o2ib10 (at 10.0.10.176@o2ib10)</pre> </div></div> master (commit fcf219d) on both server and client LU-12832

soft lockup in ldlm_bl_xx threads at read for a single shared strided file

Bug Major Resolved Not a Bug Wang Shilong Shuichi Ihara Sun, 6 Oct 2019 12:30:01 +0000 Sat, 1 Aug 2020 00:42:37 +0000 Wed, 13 Nov 2019 15:24:29 +0000 Lustre 2.13.0 0 8 <p>can you try patch <a href="https://review.whamcloud.com/#/c/36229/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36229/</a></p> <blockquote><p>can you try patch <a href="https://review.whamcloud.com/#/c/36229/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36229/</a></p></blockquote> <p>tested this patch. although it's still problem, but it was another call trace at __write_lock_failed after patch.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 7 12:45:10 c10 kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [ldlm_bl_13:31247] Oct 7 12:45:10 c10 kernel: Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_statistic xt_physdev xt_NFLOG nfnetlink_log veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ip_set_hash_ip xt_set ip_set xt_nat ipt_REJECT nf_reject_ipv4 xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) Oct 7 12:45:10 c10 kernel: mlxfw(OE) mlx4_en(OE) sunrpc iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 sg lpc_ich mei_me ioatdma mei ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad acpi_power_meter binfmt_misc knem(OE) ip_tables ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci igb crct10dif_pclmul crct10dif_common crc32c_intel mlx4_core(OE) libata ptp pps_core devlink dca i2c_algo_bit drm_panel_orientation_quirks mlx_compat(OE) [last unloaded: libcfs] Oct 7 12:45:10 c10 kernel: CPU: 10 PID: 31247 Comm: ldlm_bl_13 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-957.27.2.el7.x86_64 #1 Oct 7 12:45:10 c10 kernel: Hardware name: Intel Corporation S2600KPR/S2600KPR, BIOS SE5C610.86B.01.01.0027.071020182329 07/10/2018 Oct 7 12:45:10 c10 kernel: task: ffff9fd7a9180000 ti: ffff9fe5e8364000 task.ti: ffff9fe5e8364000 Oct 7 12:45:10 c10 kernel: RIP: 0010:[&lt;ffffffff82d88889&gt;] [&lt;ffffffff82d88889&gt;] __write_lock_failed+0x9/0x20 Oct 7 12:45:10 c10 kernel: RSP: 0018:ffff9fe5e8367c80 EFLAGS: 00000297 Oct 7 12:45:10 c10 kernel: RAX: ffff9fd56365a1d8 RBX: 0000000000000140 RCX: ffff9fd56365a1b0 Oct 7 12:45:10 c10 kernel: RDX: 0000000000000001 RSI: ffff9fd6a7258150 RDI: ffff9fd56365a1cc Oct 7 12:45:10 c10 kernel: RBP: ffff9fe5e8367c80 R08: ffff9fd6463d8bb0 R09: 0000000000000001 Oct 7 12:45:10 c10 kernel: R10: 0000000000000000 R11: ffffe16951a6ab00 R12: ffff9fd2833c1180 Oct 7 12:45:10 c10 kernel: R13: ffffffffc12ae36f R14: ffff9fe5e8367c38 R15: ffff9fd34eb66150 Oct 7 12:45:10 c10 kernel: FS: 0000000000000000(0000) GS:ffff9fd6fe880000(0000) knlGS:0000000000000000 Oct 7 12:45:10 c10 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 7 12:45:10 c10 kernel: CR2: 00000000023b6000 CR3: 000000028e610000 CR4: 00000000003607e0 Oct 7 12:45:10 c10 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 7 12:45:10 c10 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Oct 7 12:45:10 c10 kernel: Call Trace: Oct 7 12:45:10 c10 kernel: [&lt;ffffffff8316c577&gt;] _raw_write_lock+0x17/0x20 Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc0ee635a&gt;] cl_object_attr_write_lock+0x1a/0x20 [obdclass] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc1295746&gt;] osc_ldlm_blocking_ast+0x2f6/0x3a0 [osc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc10e954a&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc0ebf979&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc10f4460&gt;] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc10f897a&gt;] ldlm_cli_cancel_list_local+0xea/0x260 [ptlrpc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc10ff370&gt;] ldlm_bl_thread_main+0x5a0/0xa40 [ptlrpc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffff82ad7c40&gt;] ? wake_up_state+0x20/0x20 Oct 7 12:45:10 c10 kernel: [&lt;ffffffffc10fedd0&gt;] ? ldlm_handle_bl_callback+0x4e0/0x4e0 [ptlrpc] Oct 7 12:45:10 c10 kernel: [&lt;ffffffff82ac2e81&gt;] kthread+0xd1/0xe0 Oct 7 12:45:10 c10 kernel: [&lt;ffffffff82ac2db0&gt;] ? insert_kthread_work+0x40/0x40 Oct 7 12:45:10 c10 kernel: [&lt;ffffffff83176c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 Oct 7 12:45:10 c10 kernel: [&lt;ffffffff82ac2db0&gt;] ? insert_kthread_work+0x40/0x40 Oct 7 12:45:10 c10 kernel: Code: 00 00 0f 1f 44 00 00 41 ff e7 e8 07 00 00 00 f3 90 0f ae e8 eb f9 4c 89 3c 24 c3 90 90 90 90 90 90 90 55 48 89 e5 f0 ff 07 f3 90 &lt;83&gt; 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 90 66 2e 0f 1f 84 00 00 00 </pre> </div></div> <p>Shilong</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>it enabled kernel.softlockup_panic and we got crash and crash dump when softlockup happens on client. </p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36413" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36413</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12832" title="soft lockup in ldlm_bl_xx threads at read for a single shared strided file" class="issue-link" data-issue-key="LU-12832"><del>LU-12832</del></a> osc: invalidate kms upon blocking ast<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 92f87e3a4b25392930e3b6df9b5c6a46eb6191ff</p> <p>Just re-think the problem, IOR seems try to write file firstly stridely and in this case we might have every<br/> client full with many PW extents, more than 60K each client.</p> <p>And before Read starts, lock cancel might happen which will try to cancel tons of cache? which might make our spinlock very hot.</p> <p>So to walk around the problem, maybe use a smaller LRU SIZE shall be fine.</p> <p>But we might need add some debug information to extent tree locks to understand details etc.</p> <p>yes, fixed lru_size or short age for lru cache could be workaround, but the other hand, some application prefer keeping locks in caches as much as possible. For just an benchmark with ior for strided write/read is fine, but we need a sophisticate logic for both workload reasonably works properly <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36419" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36419</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12832" title="soft lockup in ldlm_bl_xx threads at read for a single shared strided file" class="issue-link" data-issue-key="LU-12832"><del>LU-12832</del></a> ldlm: debug extent tree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ec320272aa8d3f3b6353aff2f556b9641f9245a9</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dongyang" class="user-hover" rel="dongyang">dongyang</a> found following problem might fix the problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/ldlm/ldlm_extent.c b/lustre/ldlm/ldlm_extent.c index 9e046c91ff..36998cb5f9 100644 --- a/lustre/ldlm/ldlm_extent.c +++ b/lustre/ldlm/ldlm_extent.c @@ -890,7 +890,7 @@ static enum interval_iter ldlm_kms_shift_cb(struct interval_node *n, /* If we find a lock with a greater or equal kms, we are not the * highest lock (or we share that distinction with another lock), and * don't need to update KMS. Return old_kms and stop looking. */ - if (lock-&gt;l_policy_data.l_extent.end &gt;= arg-&gt;old_kms) { + if (lock-&gt;l_policy_data.l_extent.end + 1 &gt;= arg-&gt;old_kms) { arg-&gt;kms = arg-&gt;old_kms; arg-&gt;complete = true; RETURN(INTERVAL_ITER_STOP); </pre> </div></div> <p>The point is It is very possible that arg-&gt;old_kms = last extent end. and if we fail last one check,<br/> we will have to iterate all PR extent locks in tree which is O(N).</p> <p>It definitely seems possible that the above change could fix the problem, as the comment before <tt>ldlm_extent_shift_kms()</tt> says:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>NB: A lock on [x,y] protects a KMS of up to y + 1 bytes!</pre> </div></div> <p>. </p> <p>That said, the above-referenced code in <tt>ldlm_kms_shift_cb()</tt> hasn't changed since patch <a href="https://review.whamcloud.com/20779" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/20779</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-8272" title="Use granted extent tree to update kms" class="issue-link" data-issue-key="LU-8272"><del>LU-8272</del></a> ldlm: Use interval tree to update kms</tt>" in 2016-10-17, so it shouldn't be the <em>immediate</em> cause of a regression since the last time this code was working, so it is at least going to be a bad interaction with a patch that landed more recently. I'm not saying the above code is <b>not</b> the source of the problem, just that it would be good to know why this problem was suddenly being hit. It doesn't look like it is directly related to lock conversion, since that patch landed as v2_11_52_0-78-g541902a3f9 in 2018-07-03 before the 2.12.0 release and we haven't been seeing this problem very long.</p> <p>It looks like there are 3 patches to <tt>ldlm/ldlm_extent.c</tt> in the 2.13 development cycle:<br/> <a href="https://review.whamcloud.com/35057" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35057</a> 2019-08-27 "<tt><a href="https://jira.whamcloud.com/browse/LU-12017" title="Truncate vs setxattr deadlock with DoM" class="issue-link" data-issue-key="LU-12017"><del>LU-12017</del></a> ldlm: DoM truncate deadlock</tt>"<br/> <a href="https://review.whamcloud.com/35486" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35486</a> 2019-07-20 "<tt><a href="https://jira.whamcloud.com/browse/LU-12542" title="LDLM improvements form linux lustre client work" class="issue-link" data-issue-key="LU-12542"><del>LU-12542</del></a> ldlm: simplify lock_mode_to_index()</tt>"<br/> <a href="https://review.whamcloud.com/19665" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/19665</a> 2019-03-21 "<tt><a href="https://jira.whamcloud.com/browse/LU-8047" title="optimize ll_md_blocking_ast() for a failed lock" class="issue-link" data-issue-key="LU-8047"><del>LU-8047</del></a> llite: optimizations for not granted lock processing</tt>"</p> <p>There are also a few other patches in 2.13 that are related to KMS handling:<br/> <a href="https://review.whamcloud.com/36200" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36200</a> 2019-09-30 "<tt><a href="https://jira.whamcloud.com/browse/LU-12681" title="Data corruption - due incorrect KMS with SEL files" class="issue-link" data-issue-key="LU-12681"><del>LU-12681</del></a> osc: wrong cache of LVB attrs, part2</tt>"<br/> <a href="https://review.whamcloud.com/36199" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36199</a> 2019-09-30 "<tt><a href="https://jira.whamcloud.com/browse/LU-12681" title="Data corruption - due incorrect KMS with SEL files" class="issue-link" data-issue-key="LU-12681"><del>LU-12681</del></a> osc: wrong cache of LVB attrs</tt>"<br/> <a href="https://review.whamcloud.com/34095" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34095</a> 2019-06-25 "<tt><a href="https://jira.whamcloud.com/browse/LU-12043" title="improve Lustre single thread read performances" class="issue-link" data-issue-key="LU-12043"><del>LU-12043</del></a> llite: improve single-thread read performance</tt>"</p> <p>which would be a good place to start to see if reverting those patches, or testing the commit immediately before each patch avoids this problem. If we know the first time this problem was seen it might make it easy to track down the problematic patch.</p> <p>If there are problems tracking down the source of this bug then it would be possible to use "<tt>git bisect</tt>" at the cost of running more tests. There are currently 1008 patches from <tt>2.12.50..fcf219d</tt> so if the problem is reliably hit during testing then it would be possible to find the problem in 10 test runs or fewer. Since it is likely to be a client-side problem then it (probably) wouldn't be necessary to update the server version each time if that makes it faster for each test run.</p> <p>Yup, it is intresting to know why we did not hit the problem before, this could potentially related to LRU SIZE? or testing file size?<br/> or some locks changes regression raised the problem.</p> <p>I've tested above change with same workload, but it caused another regression at write (not read). Please see console messages whne client crashed.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[295995.970954] LustreError: 92739:0:(osc_cache.c:1153:osc_extent_make_ready()) ASSERTION( last_oap_count &gt; 0 ) failed: [295995.971011] LustreError: 92739:0:(osc_cache.c:1153:osc_extent_make_ready()) LBUG [295995.971047] Pid: 92739, comm: ldlm_bl_02 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 [295995.971048] Call Trace: [295995.971058] [&lt;ffffffffc0cb58ac&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [295995.971071] [&lt;ffffffffc0cb595c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [295995.971077] [&lt;ffffffffc123204b&gt;] osc_extent_make_ready+0x91b/0xe50 [osc] [295995.971097] [&lt;ffffffffc123499f&gt;] osc_io_unplug0+0xd9f/0x1910 [osc] [295995.971105] [&lt;ffffffffc12391e0&gt;] osc_cache_writeback_range+0x9c0/0xff0 [osc] [295995.971113] [&lt;ffffffffc1224275&gt;] osc_lock_flush+0x195/0x290 [osc] [295995.971120] [&lt;ffffffffc1224733&gt;] osc_ldlm_blocking_ast+0x2e3/0x3a0 [osc] [295995.971126] [&lt;ffffffffc107854a&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] [295995.971161] [&lt;ffffffffc1083460&gt;] ldlm_cli_cancel_local+0xa0/0x3f0 [ptlrpc] [295995.971181] [&lt;ffffffffc10891c1&gt;] ldlm_cli_cancel+0x161/0x650 [ptlrpc] [295995.971205] [&lt;ffffffffc12245ca&gt;] osc_ldlm_blocking_ast+0x17a/0x3a0 [osc] [295995.971213] [&lt;ffffffffc108d9dd&gt;] ldlm_handle_bl_callback+0xed/0x4e0 [ptlrpc] [295995.971233] [&lt;ffffffffc108e5d0&gt;] ldlm_bl_thread_main+0x800/0xa40 [ptlrpc] [295995.971252] [&lt;ffffffff932c2e81&gt;] kthread+0xd1/0xe0 [295995.971258] [&lt;ffffffff93976c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [295995.971264] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [295995.971294] Kernel panic - not syncing: LBUG </pre> </div></div> <p>I wonder if overstriping could be related. I have been enabling overstriping for this testing, but will test without overstriping in master. If the problem is still exist, we can test old commit. It takes too long time to finish this ior job without overstriping though... </p> <p><font color="#000000">background aggressive lock canceling in advance helps to avoid this problem.</font><br/> <font color="#000000">e.g. setting 30sec to lru_max_age 'lctl set_param ldlm.namespaces.*.lru_max_age=30000' </font><br/> After this change, I didn't see see problem.<br/> However, now I'm reproducing <a href="https://jira.whamcloud.com/browse/LU-12786" title="io hard read fails due to data verification fails" class="issue-link" data-issue-key="LU-12786"><del>LU-12786</del></a> again.. I will re-open <a href="https://jira.whamcloud.com/browse/LU-12786" title="io hard read fails due to data verification fails" class="issue-link" data-issue-key="LU-12786"><del>LU-12786</del></a>.</p> <p>seems to be hw issue</p> Related LU-12786 LU-13131 LU-12377 Development Rank 1|i00nov: Rank (Obsolete) 9223372036854775807 Severity