https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16655 Lustre <p>Most of the files are not accessible after 2.12 -&gt; 2.15 upgrade:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@hpcln01 ~]# ls /mnt/lustre ls: cannot access '/mnt/lustre/README': No such file or directory ls: cannot access '/mnt/lustre/other': No such file or directory ls: cannot access '/mnt/lustre/vol0': No such file or directory ls: cannot access '/mnt/lustre/fc': No such file or directory README fc other vol0 [root@hpcln01 ~]# ls -l /mnt/lustre ls: cannot access '/mnt/lustre/README': No such file or directory ls: cannot access '/mnt/lustre/other': No such file or directory ls: cannot access '/mnt/lustre/vol0': No such file or directory ls: cannot access '/mnt/lustre/fc': No such file or directory total 0 -????????? ? ? ? ? ? README d????????? ? ? ? ? ? fc d????????? ? ? ? ? ? other d????????? ? ? ? ? ? vol0 [root@hpcln01 ~]# </pre> </div></div> <p>The system had been functioning correctly before the upgrade.</p> <p>The server's kernel log contains messages like the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 20 11:33:04 hpcln02 kernel: LustreError: 1764712:0:(mdt_open.c:1519:mdt_reint_open()) hpcln-MDT0000: name 'README' present, but FID [0x200016cab:0x1e5:0x0] is invalid </pre> </div></div> LU-16655

Files not accessible after 2.12 -> 2.14/2.15 upgrade

Bug Blocker Resolved Fixed Alexander Zarochentsev Alexander Zarochentsev Wed, 22 Mar 2023 09:43:25 +0000 Mon, 29 Jan 2024 08:51:05 +0000 Tue, 18 Apr 2023 12:25:26 +0000 Lustre 2.16.0 Lustre 2.15.3 0 17 <p>The reason for the fs corruption is that information about number of OI files on the target was lost during upgrade:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osd-ldiskfs.hpcln-MDT0000.oi_scrub= name: OI_scrub magic: 0x4c5fd252 oi_files: 1 status: completed flags: param: time_since_last_completed: 547 seconds time_since_latest_start: 4525 seconds time_since_last_checkpoint: 547 seconds latest_start_position: 12 last_checkpoint_position: 4294148041 first_failure_position: N/A checked: 52509028 updated: 51670016 failed: 0 prior_updated: 0 noscrub: 38 igif: 1 success_count: 1 run_time: 3978 seconds average_speed: 13199 objects/sec real_time_speed: N/A current_position: N/A lf_scanned: 2777 lf_repaired: 0 lf_failed: 0 </pre> </div></div> <p>number of OI files became 1, i.e. the system forgot about 63 out of 64 OI files and about all FID-&gt;mapping info in those files.</p> <p>After some investigation I found a not intentioned on-disk format change, specifically in <tt>struct scrub_file</tt>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit a114f6b8c52bb97dcad093e5cb32b903ae9f602f Author: James Simmons &lt;jsimmons@infradead.org&gt; Date: Tue Mar 17 07:31:08 2020 -0400 LU-13344 servers: change request timeouts to s32 In the latest kernels time_t has been removed since time_t has been a 64 bit value just like time64_t so no need for it anymore. The field sf_run_time is stored on local media and its expected to be 32 bit in size so change it to s32 explictly. The fields the MDT coordinator that are currently time_t are a simple change to s32 since they represent a time period and not a time stamp. Lastly just update a comment in llog_swab.c since time_t will no longer exist but we want to point out time in the llog is 32 bits in size. Change-Id: I0849fc9798c191d1fed77f7bba1357a209924418 Signed-off-by: James Simmons &lt;jsimmons@infradead.org&gt; Reviewed-on: https://review.whamcloud.com/37950 Tested-by: jenkins &lt;devops@whamcloud.com&gt; Reviewed-by: Shaun Tancheff &lt;shaun.tancheff@hpe.com&gt; Reviewed-by: Andreas Dilger &lt;adilger@whamcloud.com&gt; Tested-by: Maloo &lt;maloo@whamcloud.com&gt; Reviewed-by: Oleg Drokin &lt;green@whamcloud.com&gt; diff --git a/lustre/include/lustre_scrub.h b/lustre/include/lustre_scrub.h index 38d4fb5b44..313af0f9eb 100644 --- a/lustre/include/lustre_scrub.h +++ b/lustre/include/lustre_scrub.h @@ -240,7 +240,7 @@ struct scrub_file { * to time64_t since this breaks backwards compatibility. * It shouldn't take more than 136 years to complete :-) */ - time_t sf_run_time; + s32 sf_run_time; /* How many completed OI scrub ran on the device. */ __u32 sf_success_count; ... </pre> </div></div> <p>Length of the sf_run_time field was changed without adding a padding , so the sf_oi_count field changed its offset.</p> <p>For some systems after upgrade to 2.15, an incorrect value in sf_oi_count may successfully pass sanity checks in osd_oi_init() and became new value of sf_oi_count, with consequences showed in the description of this ticket.</p> <p>The 2.14+ <tt>struct scrub_file</tt> format is actually better than the 2.12 one, because the fields are properly aligned and it does not have a hole in it. The 2.12.x struct format according to <tt>pahole</tt> is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* --- cacheline 2 boundary (128 bytes) --- */</span> time_t sf_run_time; <span class="code-comment">/* 128 8 */</span> __u32 sf_success_count; <span class="code-comment">/* 136 4 */</span> __u16 sf_oi_count; <span class="code-comment">/* 140 2 */</span> __u16 sf_internal_flags; <span class="code-comment">/* 142 2 */</span> __u32 sf_reserved_1; <span class="code-comment">/* 144 4 */</span> <span class="code-comment">/* XXX 4 bytes hole, <span class="code-keyword">try</span> to pack */</span> </pre> </div></div> <p>while the 2.14.x and later (and 2.10 and earlier, see below) format is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* --- cacheline 2 boundary (128 bytes) --- */</span> s32 sf_run_time; <span class="code-comment">/* 128 4 */</span> __u32 sf_success_count; <span class="code-comment">/* 132 4 */</span> __u16 sf_oi_count; <span class="code-comment">/* 136 2 */</span> __u16 sf_internal_flags; <span class="code-comment">/* 138 2 */</span> __u32 sf_reserved_1; <span class="code-comment">/* 140 4 */</span> </pre> </div></div> <p>Ironically, the comment in that struct says:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* How <span class="code-object">long</span> the OI scrub has run in seconds. Do NOT change * to time64_t since <span class="code-keyword">this</span> breaks backwards compatibility. * It shouldn't take more than 136 years to complete :-) */ time_t sf_run_time; </pre> </div></div> <p>Unfortunately, "time_t" is itself a signed long, so it was a 64-bit value despite what the comment says. It looks like <b>that</b> was itself changed in commit v2_10_58_0-54-g4c2f028a95 so it appears that upgrades from 2.10 to 2.12 may also have had a similar issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -237,8 +235,11 @@ struct scrub_file { <span class="code-comment">/* How many IGIF objects. */</span> __u64 sf_items_igif; - <span class="code-comment">/* How <span class="code-object">long</span> the OI scrub has run. */</span> - __u32 sf_run_time; + /* How <span class="code-object">long</span> the OI scrub has run in seconds. Do NOT change + * to time64_t since <span class="code-keyword">this</span> breaks backwards compatibility. + * It shouldn't take more than 136 years to complete :-) + */ + time_t sf_run_time; </pre> </div></div> <p>What it looks like is happening is that the <tt>sf_oi_count</tt> in memory is being shifted to read from <tt>sf_success_count</tt> on disk in the 2.12-&gt;2.14+ upgrade case. Many filesystems will never have run OI Scrub, so they will have <tt>sf_success_count=0</tt>. Some filesystems will a low number of OI Scrub runs, so "<tt>sf_oi_count = sf_success_count = 1</tt>" or similar is used. In some cases, <tt>osd_oi_init()</tt> will detect a non power-of-two value in <tt>sf_oi_count</tt> with an error like "<tt>testfs-MDT0000: Invalid OI count in scrub file 5</tt>", and reset it to 0 (which will try to read all of the <tt>oi.16.&#42;</tt> files). In the 2.10-&gt;2.12 upgrade case this bug was not noticed, since it shifted <tt>sf_internal_flags</tt> (always zero?) to <tt>sf_oi_count</tt> and all <tt>oi.16.&#42;</tt> files were read.</p> <p>On a related note, it does look like <tt>osd_oi_table_open()</tt> can trigger "<tt>LASSERT(oi_count &lt;= OSD_OI_FID_NR_MAX)</tt>" in <tt>osd_oi_table_open()</tt> if there is bad data in the <tt>OI_scrub</tt> file (e.g. upgrade system with <tt>sf_success_count &gt; 1024</tt>), so this should also be checked in <tt>osd_oi_init()</tt> and reset <tt>sf_oi_count=0</tt> in this case (and potentially any other important fields in this struct):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - <span class="code-keyword">if</span> (unlikely(sf-&gt;sf_oi_count &amp; (sf-&gt;sf_oi_count - 1)) != 0) { + <span class="code-keyword">if</span> (unlikely(sf-&gt;sf_oi_count &amp; (sf-&gt;sf_oi_count - 1) != 0 || + sf-&gt;sf_oi_count &gt; OSD_OI_FID_NR_MAX)) { sf-&gt;sf_oi_count = 0; } </pre> </div></div> <p>One option to fix this problem would be to change <tt>struct scrub_file.sf_magic</tt> to use a new value, which could be used to detect "bad" scrub files and reset the struct and start scrub from the beginning:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define SCRUB_MAGIC_V1 0x4C5FD252 #define SCRUB_MAGIC_V2 0x4C5FD253 <span class="code-object">int</span> scrub_file_load(<span class="code-keyword">const</span> struct lu_env *env, struct lustre_scrub *scrub) { <span class="code-keyword">if</span> (sf-&gt;sf_magic == SCRUB_MAGIC_V1) { sf-&gt;sf_oi_count = 0; <span class="code-comment">/* reset OI count, LU-16655 */</span> } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (sf-&gt;sf_magic != SCRUB_MAGIC_V2) { CDEBUG(D_LFSCK, <span class="code-quote">"%s: invalid scrub magic 0x%x != 0x%x\n"</span>, scrub-&gt;os_name, sf-&gt;sf_magic, SCRUB_MAGIC_V2); <span class="code-keyword">return</span> -EFAULT; } </pre> </div></div> <p>It seems possible to "fix" old <tt>OI_scrub</tt> files by just resetting the "<tt>sf_oi_scrub=0</tt>" if <tt>SCRUB_MAGIC_V1</tt> is found.</p> <p>The <tt>OI_scrub</tt> file itself is only useful for temporarily storing the scrub state so that scrub doesn't start from inode 1 on each mount. This is useful if scrub is taking a long time to complete, or the server crashes in the middle. Resetting the scrub state on an upgrade is totally fine and would avoid this issue.</p> <p>Alternately, or in addition, it makes sense for <tt>osd_oi_table_open()</tt> to not blindly trust <tt>sf_oi_count</tt> and try to access at least the default number of OI files like <tt>osd_oi_probe()</tt> so that it doesn't do something similarly bad to the OI files if the <tt>OI_scrub</tt> file is corrupted on disk in some other way.</p> <p>Since this is an on-disk structure it should be checked in wirecheck/wiretest to avoid such breakage in the future. It may also make sense to make "<tt>struct scrub_file</tt>" a fixed size (e.g. 1KB) and also store the magic at the end to be able to detect other kinds of corruption.</p> <p>"Alexander Zarochentsev &lt;alexander.zarochentsev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50455" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50455</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: upgrade scrub_file from 2.12 format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc6debf1a352611a22e9024443d9422c42f3e64d</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50480" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50480</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: upgrade scrub_file from 2.12 format<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: e0359d4746e54daeab01f9f173a3db3d42476adc</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50482" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50482</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> build: add wirecheck/wiretest for disk structs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 357f65dedeb6637171d1e7bc9da2591e9a15d66c</p> <p>The issue related to the change of sf_run_time format between 2.10 to 2.12 (from s32 to time_t) was:</p> <ul class="alternate" type="square"> <li><a href="https://jira.whamcloud.com/browse/LU-14864" title="osd_fid_lookup() ASSERTION( !updated ) failed" class="issue-link" data-issue-key="LU-14864">LU-14864</a>: osd_fid_lookup() ASSERTION( !updated ) failed</li> </ul> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50496" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50496</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: change sf_uuid to guid_t<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ccc72993aaddbd42bb2692848784690086b8e241</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50480/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50480/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: upgrade scrub_file from 2.12 format<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: a209505361cf858ec91b0a89992691c5b005ab72</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50455/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50455/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: upgrade scrub_file from 2.12 format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 126275ba8339540e46f1c517decd3d69ad1cc42c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50496/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50496/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: change sf_uuid to guid_t<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 20621c761b787cd7b08a9d0abc0fc362f0bb0122</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50482/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50482/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> build: add wirecheck/wiretest for disk structs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a3a713c8e1a47edc625963092b6b3f27079cdf2d</p> <p>Landed for 2.16</p> Related LU-14864 LU-16677 LU-16689 LU-16873 Development Rank 1|i03gvr: Rank (Obsolete) 9223372036854775807 Severity