https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16152 Lustre <p>When applying a YAML layout file with extents of 2147483648 or larger, these extents appear as 18446744071562067968. Example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lfs setstripe -E 2048M -c 4 -E EOF -c 8 testdir # lfs getstripe --yaml -d testdir &gt; 2048M.lyl # cat 2048M.lyl   lcm_layout_gen:    0   lcm_mirror_count:  1   lcm_entry_count:   2   component0:     lcme_id:             N/A     lcme_mirror_id:      N/A     lcme_flags:          0     lcme_extent.e_start: 0     lcme_extent.e_end:   2147483648     sub_layout:       stripe_count:  4       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1  component1:     lcme_id:             N/A     lcme_mirror_id:      N/A     lcme_flags:          0     lcme_extent.e_start: 2147483648     lcme_extent.e_end:   EOF     sub_layout:       stripe_count:  8       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1 # mkdir tst_2048M # lfs setstripe --yaml 2048M.lyl tst_2048M # lfs getstripe -d tst_2048M   lcm_layout_gen:    0   lcm_mirror_count:  1   lcm_entry_count:   2     lcme_id:             N/A     lcme_mirror_id:      N/A     lcme_flags:          0     lcme_extent.e_start: 0     lcme_extent.e_end:   18446744071562067968       stripe_count:  4       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1    lcme_id:             N/A     lcme_mirror_id:      N/A     lcme_flags:          0     lcme_extent.e_start: 18446744071562067968     lcme_extent.e_end:   EOF       stripe_count:  8       stripe_size:   1048576       pattern:       raid0       stripe_offset: -1</pre> </div></div> <p>Using "lfs setstripe --copy testdir" instead of "lfs setstripe --yaml 2048M.lyl" works as intended. Ending the first component at 2047M works with either method.</p> <p>Unfortunately, I did not catch this in time and several files were restriped with similar insane layouts. Attempts to re-stripe them properly occasionally trigger kernel panics on the metadata server. Here is one of the early ones, which would happen immediately after the MDS recovered after reboot.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [Aug31 20:55] Lustre: DFS-L-MDT0000: Denying connection <span class="code-keyword">for</span> <span class="code-keyword">new</span> client 65a07c20-0fc9-26df-0102-6dd1be2412e7 (at 10.201.32.11@o2ib1), waiting <span class="code-keyword">for</span> 369 known clients (329 recovered, 6 in progress, and 0 evicted) to recover in 4:19 [ +13.335095] Lustre: DFS-L-MDT0000: Recovery over after 0:54, of 369 clients 369 recovered and 0 were evicted. [  +0.001392] LustreError: 41361:0:(osd_io.c:311:kmem_to_page()) ASSERTION( !((unsigned <span class="code-object">long</span>)addr &amp; ~(~(((1UL) &lt;&lt; 12)-1))) ) failed:  [  +0.000079] LustreError: 41361:0:(osd_io.c:311:kmem_to_page()) LBUG [  +0.000038] Pid: 41361, comm: mdt_io00_002 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 [  +0.000001] Call Trace: [  +0.000013]  [&lt;ffffffffc0b647cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [  +0.000015]  [&lt;ffffffffc0b6487c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [  +0.000008]  [&lt;ffffffffc11d3a7c&gt;] osd_zap_lookup.isra.15.part.16+0x0/0x36 [osd_zfs] [  +0.000017]  [&lt;ffffffffc11b845f&gt;] osd_bufs_get+0x5ff/0xf80 [osd_zfs] [  +0.000011]  [&lt;ffffffffc1361389&gt;] mdt_obd_preprw+0xd09/0x10a0 [mdt] [  +0.000032]  [&lt;ffffffffc103365e&gt;] tgt_brw_read+0xa1e/0x1ed0 [ptlrpc] [  +0.000095]  [&lt;ffffffffc1031eea&gt;] tgt_request_handle+0xada/0x1570 [ptlrpc] [  +0.000058]  [&lt;ffffffffc0fd6bcb&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [  +0.000047]  [&lt;ffffffffc0fda534&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] [  +0.000044]  [&lt;ffffffffa2ec5e61&gt;] kthread+0xd1/0xe0 [  +0.000008]  [&lt;ffffffffa3595df7&gt;] ret_from_fork_nospec_end+0x0/0x39 [  +0.000008]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [  +0.000026] Kernel panic - not syncing: LBUG [  +0.000030] CPU: 22 PID: 41361 Comm: mdt_io00_002 Tainted: P           OE  ------------   3.10.0-1160.49.1.el7.x86_64 #1 [  +0.000059] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 [  +0.000043] Call Trace: [  +0.000021]  [&lt;ffffffffa3583539&gt;] dump_stack+0x19/0x1b [  +0.000042]  [&lt;ffffffffa357d241&gt;] panic+0xe8/0x21f [  +0.000037]  [&lt;ffffffffc0b648cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [  +0.000050]  [&lt;ffffffffc11d3a7c&gt;] kmem_to_page.part.16+0x36/0x36 [osd_zfs] [  +0.000052]  [&lt;ffffffffc11b845f&gt;] osd_bufs_get+0x5ff/0xf80 [osd_zfs] [  +0.000056]  [&lt;ffffffffc1361389&gt;] mdt_obd_preprw+0xd09/0x10a0 [mdt] [  +0.000085]  [&lt;ffffffffc103365e&gt;] tgt_brw_read+0xa1e/0x1ed0 [ptlrpc] [  +0.000082]  [&lt;ffffffffc0d1df29&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [  +0.000087]  [&lt;ffffffffc1001cd6&gt;] ? null_alloc_rs+0x186/0x340 [ptlrpc] [  +0.000080]  [&lt;ffffffffc0fc9985&gt;] ? lustre_pack_reply_v2+0x135/0x290 [ptlrpc] [  +0.000084]  [&lt;ffffffffc0fc9b4f&gt;] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [  +0.000080]  [&lt;ffffffffc0fc9cd1&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [  +0.000088]  [&lt;ffffffffc1031eea&gt;] tgt_request_handle+0xada/0x1570 [ptlrpc] [  +0.000085]  [&lt;ffffffffc100b601&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [  +0.000052]  [&lt;ffffffffc0b64bde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [  +0.000080]  [&lt;ffffffffc0fd6bcb&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [  +0.000082]  [&lt;ffffffffc0fd36e5&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [  +0.000042]  [&lt;ffffffffa2ed3233&gt;] ? __wake_up+0x13/0x20 [  +0.000070]  [&lt;ffffffffc0fda534&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] [  +0.000077]  [&lt;ffffffffc0fd9a00&gt;] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [  +0.000045]  [&lt;ffffffffa2ec5e61&gt;] kthread+0xd1/0xe0 [  +0.000032]  [&lt;ffffffffa2ec5d90&gt;] ? insert_kthread_work+0x40/0x40 [  +0.000048]  [&lt;ffffffffa3595df7&gt;] ret_from_fork_nospec_begin+0x21/0x21 [  +0.000040]  [&lt;ffffffffa2ec5d90&gt;] ? insert_kthread_work+0x40/0x40 [  +0.000042] ------------[ cut here ]------------ [  +0.000032] WARNING: CPU: 26 PID: 6640 at arch/x86/kernel/smp.c:127 native_smp_send_reschedule+0x65/0x70 [  +0.000042] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) crct10dif_generic ksocklnd(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace libcfs(OE) fscache iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi zfs(POE) mgag200 i2c_algo_bit ttm kvm drm_kms_helper irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel syscopyarea ghash_clmulni_intel sysfillrect sysimgblt fb_sys_fops aesni_intel lrw gf128mul glue_helper zunicode(POE) ablk_helper cryptd drm zlua(POE) pcspkr zcommon(POE) znvpair(POE) zavl(POE) icp(POE) lpc_ich drm_panel_orientation_quirks mei_me mei ib_iser spl(OE) libiscsi scsi_transport_iscsi ses enclosure sg wmi [  +0.000434]  acpi_power_meter rpcrdma sunrpc ib_ipoib rdma_ucm ib_umad rdma_cm ib_cm iw_cm sch_fq ip_tables ipmi_si ipmi_devintf ipmi_msghandler mlx5_ib ib_uverbs ib_core mlx5_core mlxfw devlink mpt3sas raid_class scsi_transport_sas tg3 ptp pps_core ahci libahci libata sd_mod crc_t10dif crct10dif_common [  +0.000164] CPU: 26 PID: 6640 Comm: z_wr_iss Tainted: P           OE  ------------   3.10.0-1160.49.1.el7.x86_64 #1 [  +0.000046] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 [  +0.001264] Call Trace: [  +0.001247]  [&lt;ffffffffa3583539&gt;] dump_stack+0x19/0x1b [  +0.001259]  [&lt;ffffffffa2e9b278&gt;] __warn+0xd8/0x100 [  +0.001253]  [&lt;ffffffffa2e9b3bd&gt;] warn_slowpath_null+0x1d/0x20 [  +0.001226]  [&lt;ffffffffa2e59495&gt;] native_smp_send_reschedule+0x65/0x70 [  +0.001200]  [&lt;ffffffffa2edac9e&gt;] try_to_wake_up+0x2fe/0x390 [  +0.001188]  [&lt;ffffffffa2edadab&gt;] wake_up_q+0x5b/0x80 [  +0.001177]  [&lt;ffffffffa318f18b&gt;] rwsem_wake+0x8b/0xe0 [  +0.001156]  [&lt;ffffffffa31981eb&gt;] call_rwsem_wake+0x1b/0x30 [  +0.001140]  [&lt;ffffffffc08505e5&gt;] </pre> </div></div> <p>Killing off all processes on the clients stuck on IO, then re-mounting the filesystem allowed the metadata server to stay up. A "lctl lfsck_start -A -c -C -o" completed with no major issues, and scrubbing the zpools also completed successfully, so I don't think data was lost.</p> <p>Searching this Jira and the Internet did not get me any relevant hits for either the PFL YAML issue or failed assertion.</p> Servers: CentOS 7.9 (3.10.0-1160.49.1.el7.x86_64), Lustre 2.12.8, ZFS 0.8.5 <br/> 2.12 Clients: CentOS 7.9 (3.10.0-1160.53.1.el7.x86_64), Lustre 2.12.8 <br/> 2.15 Clients: CentOS 7.9 (3.10.0-1160.76.1.el7.x86_64), Lustre 2.15.1 LU-16152

PFL YAML file with extent >= 2G leads to overflow when used as template; may trigger MDS kernel panic

Bug Major Resolved Fixed Feng Lei Nathan Crawford Wed, 14 Sep 2022 00:07:10 +0000 Thu, 24 Aug 2023 00:41:07 +0000 Wed, 19 Jul 2023 02:50:59 +0000 Lustre 2.12.8 Lustre 2.15.1 Lustre 2.16.0 0 6 <p>Feng, can you please take a look. Probably int vs long parsing or similar?</p> <p>In addition to fixing the YAML parsing, the MDS should also be modified so that it does a sanity check on this field and does not crash on bad data from disk or the network. </p> <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48649" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48649</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16152" title="PFL YAML file with extent &gt;= 2G leads to overflow when used as template; may trigger MDS kernel panic" class="issue-link" data-issue-key="LU-16152"><del>LU-16152</del></a> utils: fix integer overflow in cYAML parser<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 65ec793b1cbdce477d288865e5d4f0ac3d071365</p> <p>The patch above fixes only the cYAML parser. Need another patch for param checking in kernel.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> </p> <p>Is it OK to define any negative value except EOF (-1) (and other pre-defined special values?) as invalid for extent start/end (if covert them to s64)?</p> <p>Yes, the negative offsets should be considered invalid. With the YAML parser fixed, are there other sources of invalid start/end offsets?</p> <p>It's hard to know where the invalid data is from. The lu_extent.e_start and lu_extent.e_end are both u64 type, so there is no invalid u64 data unless we define some data (e.g., s64 negative values except -1) as invalid on MDS.</p> <p>Is it possible to recover any files that have been migrated with these bad extents? Attempts to read the files, or use lfs_migrate/lfs migrate to restripe them with something sensible triggers the kernel panic on the MDS immediately. Manually deleting the problematic components (when not initialized yet), then "lfs migrate -c 1 badfile" leads to errors such as: "lfs migrate: cannot get group lock: Invalid argument (22)"</p> <p>Verbose striping info on bad file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>composite_header:   lcm_magic:         0x0BD60BD0   lcm_size:          552   lcm_flags:         0   lcm_layout_gen:    2   lcm_mirror_count:  1   lcm_entry_count:   5 components:   - lcme_id:             131073     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 0     lcme_extent.e_end:   131072     lcme_offset:         272     lcme_size:           32     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000b836       lmm_object_id:     0x19a       lmm_fid:           [0x20000b836:0x19a:0x0]       lmm_stripe_count:  0       lmm_stripe_size:   131072       lmm_pattern:       mdt       lmm_layout_gen:    0       lmm_stripe_offset: 0   - lcme_id:             131074     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 131072     lcme_extent.e_end:   16777216     lcme_offset:         304     lcme_size:           56     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000b836       lmm_object_id:     0x19a       lmm_fid:           [0x20000b836:0x19a:0x0]       lmm_stripe_count:  1       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 6       lmm_objects:       - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2de85:0x0] }   - lcme_id:             131075     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 16777216     lcme_extent.e_end:   1073741824     lcme_offset:         360     lcme_size:           80     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000b836       lmm_object_id:     0x19a       lmm_fid:           [0x20000b836:0x19a:0x0]       lmm_stripe_count:  2       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 2       lmm_objects:       - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x861394f:0x0] }       - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x12e4858e:0x0] }   - lcme_id:             131076     lcme_mirror_id:      2     lcme_flags:          0     lcme_extent.e_start: 1073741824     lcme_extent.e_end:   18446744071562067968     lcme_offset:         440     lcme_size:           56     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000b836       lmm_object_id:     0x19a       lmm_fid:           [0x20000b836:0x19a:0x0]       lmm_stripe_count:  4       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: -1   - lcme_id:             131077     lcme_mirror_id:      2     lcme_flags:          0     lcme_extent.e_start: 18446744071562067968     lcme_extent.e_end:   EOF     lcme_offset:         496     lcme_size:           56     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000b836       lmm_object_id:     0x19a       lmm_fid:           [0x20000b836:0x19a:0x0]       lmm_stripe_count:  8       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: -1 </pre> </div></div> <p>After "lfs setstripe --comp-del -I 131077 badfile; lfs setstripe --comp-del -I 131076 badfile"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>  lcm_layout_gen:    4   lcm_mirror_count:  1   lcm_entry_count:   3     lcme_id:             131073     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 0     lcme_extent.e_end:   131072       lmm_stripe_count:  0       lmm_stripe_size:   131072       lmm_pattern:       mdt       lmm_layout_gen:    0       lmm_stripe_offset: 0     lcme_id:             131074     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 131072     lcme_extent.e_end:   16777216       lmm_stripe_count:  1       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 6       lmm_objects:       - 0: { l_ost_idx: 6, l_fid: [0x100060000:0x2de85:0x0] }     lcme_id:             131075     lcme_mirror_id:      2     lcme_flags:          init     lcme_extent.e_start: 16777216     lcme_extent.e_end:   1073741824       lmm_stripe_count:  2       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 2       lmm_objects:       - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x861394f:0x0] }       - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x12e4858e:0x0] } </pre> </div></div> <p>Alternately, is there a way to disable this assertion for long enough to migrate the files back into shape?</p> <p> </p> <p>Nathan, the patch <a href="https://review.whamcloud.com/48684" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48684</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16194" title="Define negative PFL extent start/end as invalid" class="issue-link" data-issue-key="LU-16194">LU-16194</a> lod: define negative extent offset as invalid</tt>" should stop the MDS from crashing, but it will currently mark the whole file as having an invalid layout and the files would be inaccessible It may be possible to change the code temporarily so that <tt>e_start/e_end = 0xffffffffnnnnnnnn</tt> are interpreted as <tt>0xnnnnnnnn</tt>, but this hasn't been implemented yet.</p> <p>I left some notes in that patch so you could potentially make a patch to fix this yourself, I don't think it would be too complex. This would both allow you to migrate the affected files without crashing (it would "fix" the layouts in memory only), and ideally be a proper patch that could be landed in case anyone else is affected by this bug before the YAML import fix is widely deployed.</p> <p>Thanks! Will attempt and report.</p> <p>Attempted to patch as in referenced <a href="https://jira.whamcloud.com/browse/LU-16194" title="Define negative PFL extent start/end as invalid" class="issue-link" data-issue-key="LU-16194">LU-16194</a>, but kernel panic remains. I'm attaching my patches against 2.12.8 to <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/46279/46279_lod_lov.c.patch" title="lod_lov.c.patch attached to LU-16152">lod_lov.c.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>, <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/46280/46280_lod_object.c.patch" title="lod_object.c.patch attached to LU-16152">lod_object.c.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>, and <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/46281/46281_lov_ea.c.patch" title="lov_ea.c.patch attached to LU-16152">lov_ea.c.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>. </p> <p>Notes:<br/>   – I'm not familiar with best practices for CWARN messages and don't know what useful info should be included. Made some guesses.<br/>   – In the few seconds the MDS was up before panicking, I saw some of the CWARN message come though from both lod_lov.c and lod_object.c. I believe that they were interpreting extent.e_end=eof as "-1", then converting it to 4G. The extent-end-checking part needs to handle this; I'll try to rig something up.<br/>   – The lov_ea.c patch compiles, but was not tested. I don't know how to get the FID into the error message. Is there anything in the scope of lsm_unpackmd_comp_md_v1() that is recommended to use?</p> <p>I'm going to spin up a single-server, single-client system to try to reproduce the bug and test fixes. For now, I've set the subdirectory that was migrated to 000 permissions. I believe the problem files are all there.</p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=nathan.crawford%40uci.edu" class="user-hover" rel="nathan.crawford@uci.edu">nathan.crawford@uci.edu</a> </p> <p>I guess LUSTRE_EOF should be excluded, so the condition should be like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> if (lsme-&gt;lsme_extent.e_start != LUSTRE_EOF &amp;&amp; (lsme-&gt;lsme_extent.e_start &gt;&gt; 32) == 0xffffffffULL) </pre> </div></div> <p>Nathan, I grabbed your patches yesterday and was working on a patch that I could push to Gerrit instead of as attachments here. There were a few changes that I'd made from your patches - made a <tt>LOD_COMP_EXT_BAD</tt> constant, moved all of the extent checks into a helper function instead of duplicating the code multiple times. </p> <p>One question I am had wa which Lustre version your patch was against? One of the patches didn't apply and the code didn't look very similar. </p> <p>Feng Lei, it isn't clear whether there is a valid case where <tt>e_start == LUSTRE_EOF</tt> is ever valid?</p> <p>No <tt>e_start == LUSTRE_EOF</tt>. I mean do not convert LUSTRE_EOF from <tt>0xffff ffff ffff ffff</tt> to <tt>0x0000 0000 ffff ffff</tt> for <tt>e_end</tt>;</p> <p>Andreas, the patches were generated against 2.12.8 with commands like "git diff 2.12.8 lustre/lov/lov_ea.c". The actual version of lustre is 6 commits past 2.12.8 on b2_12 (up to 5457c37ec9f76e2fb1656c29848412522dbb81fd, "<a href="https://jira.whamcloud.com/browse/LU-15292" title="kernel update [RHEL7.9 3.10.0-1160.49.1.el7]" class="issue-link" data-issue-key="LU-15292"><del>LU-15292</del></a> kernel: kernel update RHEL7.9 <span class="error">&#91;3.10.0-1160.49.1.el7&#93;</span>", 2 Dec 2021). The only other modifications were fixes to lustre-dkms_pre-build.sh to cope with an output format change of "dkms status", and to config/lustre-build-zfs.m4 to handle when multiple subdirectories of /usr/src match "&#42;zfs&#42;".</p> <p>Regarding the kernel panics on the MDS, I have NOT been able to reproduce them on a single-server test system. I can generate files with the bad PFL, read them, copy them, restripe them with lfs_migrate, etc.. </p> <p>Also confusing: some files on the original problem file system that previously had bad extent values do not now, but still panic when accessed.</p> <p>I will try a few more permutations of setting bad layouts and restriping on the test system to see if I can re-create the panic.</p> <p>An example bad 20K file (with reasonable extent values):</p> <p>du --apparent-size  badfile<br/> 20</p> <p>but the output from "lfs getstripe -v badfile" looks like it belongs to a different file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>composite_header:   lcm_magic:         0x0BD60BD0   lcm_size:          744   lcm_flags:         0   lcm_layout_gen:    6   lcm_mirror_count:  1   lcm_entry_count:   5 components:   - lcme_id:             1     lcme_mirror_id:      0     lcme_flags:          init     lcme_extent.e_start: 0     lcme_extent.e_end:   131072     lcme_offset:         272     lcme_size:           32     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000ec94       lmm_object_id:     0x1defe       lmm_fid:           [0x20000ec94:0x1defe:0x0]       lmm_stripe_count:  0       lmm_stripe_size:   131072       lmm_pattern:       mdt       lmm_layout_gen:    0       lmm_stripe_offset: 0   - lcme_id:             2     lcme_mirror_id:      0     lcme_flags:          init     lcme_extent.e_start: 131072     lcme_extent.e_end:   16777216     lcme_offset:         304     lcme_size:           56     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000ec94       lmm_object_id:     0x1defe       lmm_fid:           [0x20000ec94:0x1defe:0x0]       lmm_stripe_count:  1       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 1       lmm_objects:       - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x76112e4:0x0] }   - lcme_id:             3     lcme_mirror_id:      0     lcme_flags:          init     lcme_extent.e_start: 16777216     lcme_extent.e_end:   1073741824     lcme_offset:         360     lcme_size:           80     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000ec94       lmm_object_id:     0x1defe       lmm_fid:           [0x20000ec94:0x1defe:0x0]       lmm_stripe_count:  2       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 2       lmm_objects:       - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x74e2831:0x0] }       - 1: { l_ost_idx: 3, l_fid: [0x100030000:0xa0cadca:0x0] }   - lcme_id:             4     lcme_mirror_id:      0     lcme_flags:          init     lcme_extent.e_start: 1073741824     lcme_extent.e_end:   34359738368     lcme_offset:         440     lcme_size:           128     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000ec94       lmm_object_id:     0x1defe       lmm_fid:           [0x20000ec94:0x1defe:0x0]       lmm_stripe_count:  4       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 3       lmm_objects:       - 0: { l_ost_idx: 3, l_fid: [0x100030000:0xa0cadcb:0x0] }       - 1: { l_ost_idx: 5, l_fid: [0x100050000:0x115f8373:0x0] }       - 2: { l_ost_idx: 2, l_fid: [0x100020000:0x74e2832:0x0] }       - 3: { l_ost_idx: 1, l_fid: [0x100010000:0x76112e5:0x0] }   - lcme_id:             5     lcme_mirror_id:      0     lcme_flags:          init     lcme_extent.e_start: 34359738368     lcme_extent.e_end:   EOF     lcme_offset:         568     lcme_size:           176     sub_layout:       lmm_magic:         0x0BD10BD0       lmm_seq:           0x20000ec94       lmm_object_id:     0x1defe       lmm_fid:           [0x20000ec94:0x1defe:0x0]       lmm_stripe_count:  6       lmm_stripe_size:   1048576       lmm_pattern:       raid0       lmm_layout_gen:    0       lmm_stripe_offset: 0       lmm_objects:       - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6c01b46:0x0] }       - 1: { l_ost_idx: 3, l_fid: [0x100030000:0xa0cadcc:0x0] }       - 2: { l_ost_idx: 1, l_fid: [0x100010000:0x76112e6:0x0] }       - 3: { l_ost_idx: 5, l_fid: [0x100050000:0x115f8374:0x0] }       - 4: { l_ost_idx: 4, l_fid: [0x100040000:0xa4c5a4d:0x0] }       - 5: { l_ost_idx: 2, l_fid: [0x100020000:0x74e2833:0x0] } </pre> </div></div> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49065" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49065</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16152" title="PFL YAML file with extent &gt;= 2G leads to overflow when used as template; may trigger MDS kernel panic" class="issue-link" data-issue-key="LU-16152"><del>LU-16152</del></a> lov: handle negative PFL layout offsets<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: baf015a9582594c9b8c3589d54f18f60fdb04f34</p> <p>Nathan, I've pushed what I <em>think</em> is a reasonable update of your patches into Gerrit, but it is totally untested. Please take a look. Note that this patch is on master, so you might need to massage it a bit to apply to b2_12.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48649/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48649/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16152" title="PFL YAML file with extent &gt;= 2G leads to overflow when used as template; may trigger MDS kernel panic" class="issue-link" data-issue-key="LU-16152"><del>LU-16152</del></a> utils: fix integer overflow in cYAML parser<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 53b3d1de99277f8d3e4af475986802ee3dcdd304</p> Related LU-16194 Development Rank 1|i0302n: Rank (Obsolete) 9223372036854775807 Severity