https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16689 Lustre <p>After upgrading filesystem from 2.12 to 2.15.2 Several top level directories got corrupted. </p> <p><span class="error">&#91;root@nbp11-srv1 ~&#93;</span># ls -l /nobackupp11/<br/> ls: cannot access '/nobackupp11/ylin4': No such file or directory<br/> ls: cannot access '/nobackupp11/mbarad': No such file or directory<br/> ls: cannot access '/nobackupp11/ldgrant': No such file or directory<br/> ls: cannot access '/nobackupp11/kknizhni': No such file or directory<br/> ls: cannot access '/nobackupp11/mzhao4': No such file or directory<br/> ls: cannot access '/nobackupp11/afahad': No such file or directory<br/> ls: cannot access '/nobackupp11/jliu7': No such file or directory<br/> ls: cannot access '/nobackupp11/jswest': No such file or directory<br/> ls: cannot access '/nobackupp11/hsp': No such file or directory<br/> ls: cannot access '/nobackupp11/vjespos1': No such file or directory<br/> ls: cannot access '/nobackupp11/ssepka': No such file or directory<br/> ls: cannot access '/nobackupp11/cjang1': No such file or directory</p> <p> </p> <p>debugfs:  stat ylin4<br/> Inode: 43051102   Type: directory    Mode:  0000   Flags: 0x80000<br/> Generation: 503057142    Version: 0x00000000:00000000<br/> User:     0   Group:     0   Project:     0   Size: 4096<br/> File ACL: 0<br/> Links: 2   Blockcount: 8<br/> Fragment:  Address: 0    Number: 0    Size: 0<br/>  ctime: 0x63dd8e2f:22c83c08 &#8211; Fri Feb  3 14:43:59 2023<br/>  atime: 0x63dd8e2f:22c83c08 &#8211; Fri Feb  3 14:43:59 2023<br/>  mtime: 0x63dd8e2f:22c83c08 &#8211; Fri Feb  3 14:43:59 2023<br/> crtime: 0x63dd8e2f:22c83c08 &#8211; Fri Feb  3 14:43:59 2023<br/> Size of extra inode fields: 32<br/> Extended attributes:<br/>   lma: fid=<span class="error">&#91;0x280015902:0x2:0x0&#93;</span> compat=0 incompat=2<br/> EXTENTS:<br/> (0):671099035</p> <p> </p> <p>Not thinking I delete these via ldiskfs. The data is still there how can we recover the director data.</p> <ol> <li>lfs quota -u ylin4  /nobackupp11<br/> Disk quotas for usr ylin4 (uid 11560):<br/>      Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace<br/>    /nobackupp11 11337707848* 1073741824 2147483648       -  208359  500000  600000       -</li> </ol> <p> </p> <p> </p> LU-16689

upgrade to 2.15.2 lost sever top level directories

Bug Major Resolved Duplicate Andreas Dilger Mahmoud Hanafi Thu, 30 Mar 2023 18:58:30 +0000 Sat, 13 May 2023 14:40:40 +0000 Sat, 13 May 2023 14:40:19 +0000 0 4 <p>I started a lfsck dry-run</p> <p>on MDT0 getting a lot of these errors that are for files with hard links</p> <p>Mar 30 12:32:28 nbp11-srv1 kernel: ret_from_fork+0x1f/0x40<br/> Mar 30 12:32:28 nbp11-srv1 kernel: Lustre: nbp11-MDT0000-osd: namespace LFSCK add flags for <span class="error">&#91;0x20004ca8c:0x8986:0x0&#93;</span> in the trace file, flags 1, old 0, new 1: rc = -22<br/> Mar 30 12:32:28 nbp11-srv1 kernel: CPU: 34 PID: 1520983 Comm: lfsck Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-425.3.1.el8_lustre.x86_64 #1<br/> Mar 30 12:32:28 nbp11-srv1 kernel: Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 04/21/2022<br/> Mar 30 12:32:28 nbp11-srv1 kernel: Call Trace:<br/> Mar 30 12:32:28 nbp11-srv1 kernel: dump_stack+0x41/0x60<br/> Mar 30 12:32:28 nbp11-srv1 kernel: lfsck_trans_create.part.58+0x63/0x70 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: lfsck_namespace_trace_update+0x972/0x980 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: lfsck_namespace_exec_oit+0x87d/0x970 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: lfsck_master_oit_engine+0xc56/0x1360 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: lfsck_master_engine+0x512/0xcd0 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: ? __schedule+0x2d9/0x860<br/> Mar 30 12:32:28 nbp11-srv1 kernel: ? finish_wait+0x80/0x80<br/> Mar 30 12:32:28 nbp11-srv1 kernel: ? lfsck_master_oit_engine+0x1360/0x1360 <span class="error">&#91;lfsck&#93;</span><br/> Mar 30 12:32:28 nbp11-srv1 kernel: kthread+0x10a/0x120<br/> Mar 30 12:32:28 nbp11-srv1 kernel: ? set_kthread_struct+0x50/0x50<br/> Mar 30 12:32:28 nbp11-srv1 kernel: ret_from_fork+0x1f/0x40</p> <p>on MDT2 getting these errors</p> <p>Mar 30 12:33:43 nbp11-srv5 kernel: Lustre: nbp11-MDT0002-osd: layout LFSCK master found bad lmm_oi for <span class="error">&#91;0x2400ecb78:0x1e8bb:0x0&#93;</span>: rc = 56<br/> Mar 30 12:33:43 nbp11-srv5 kernel: Lustre: nbp11-MDT0002-osd: layout LFSCK master found bad lmm_oi for <span class="error">&#91;0x2400ecb78:0x1e8bc:0x0&#93;</span>: rc = 56</p> <p>These are the files for the bad directories.</p> <p>I recovered the files.</p> <p>I found the parent fid and cd into /fs/.lustre/fid/fidnum then just move all contents to a newly created directory</p> <p>I still like to understand what caused the corruption. </p> <p>Hi Mahmoud, 2 questions:<br/> What does stat look like on nobackupp11 via debugfs?<br/> How did you find ylin4 in debugfs?</p> <p>This looks like <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a>, which was caused by a bad code change breaking the on-disk file format for OI Scrub. If Scrub has been run on a filesystem prior to upgrade then it will incorrectly read the fields from this file. The patch <a href="https://review.whamcloud.com/50455" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50455</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> scrub: upgrade scrub_file from 2.12 format</tt>" fixes this issue and <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> describes the details (though it is too late to avoid this bug for your system).</p> <p>It may be that mounting the MDT with "<tt>-o resetoi</tt>" would have rebuilt the OI files without having to move them from <tt>lost+found</tt>, in case someone finds this ticket in the future.</p> <p>I used debugfs to dump all fid in /REMOTE_DIR on each MDT. Then I did a lookup of the fid2path to match the directories that were missing. I then cd into the /fs/.lustre/fid/fidnum and moved all contents to its new location.</p> <p>Dry-run lfsck still running and finding lots of these<br/> Mar 30 12:53:39 nbp11-srv5 kernel: Lustre: nbp11-MDT0002-osd: layout LFSCK master found bad lmm_oi for <span class="error">&#91;0x2400ecb98:0x8467:0x0&#93;</span>: rc = 56<br/> Mar 30 12:53:39 nbp11-srv5 kernel: Lustre: nbp11-MDT0002-osd: layout LFSCK master found bad lmm_oi for <span class="error">&#91;0x2400ecb98:0x8468:0x0&#93;</span>: rc = 56</p> <p>These are files under the directories that gotten corrupted. </p> <p>Mahmoud, do you have any logs from the mount after the upgrade that indicate OI Scrub has been run/completed on the MDTs? It would be worthwhile to check the state of the OI files on the MDTs to confirm that they are correct:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mds# lctl get_param osd-ldiskfs.*.oi_scrub osd-ldiskfs.testfs-MDT0000.oi_scrub= name: OI_scrub magic: 0x4c5fd252 oi_files: 64 status: completed flags: param: time_since_last_completed: 6 seconds time_since_latest_start: 6 seconds time_since_last_checkpoint: 6 seconds : </pre> </div></div> <p>The important information here is that it reports <tt>oi_files: 64</tt> and not some other number (which is what the <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> bug broke). The OI files map the Lustre FIDs to local inode numbers, so without these most of the by-FID lookups will be broken. The OI Scrub can rebuild the OI files from the FID xattr stored in each inode.</p> <p>If this is showing "<tt>oi_files: 1</tt>" or 2 or similar, my recommendation would be to mount the MDTs with "<tt>-o resetoi</tt>" to force a rebuild of the OI files, or alternately mount MDTs with ldiskfs and move the "<tt>oi.16.X</tt>" files out of the filesystem and then remount as Lustre and it should rebuild them automatically at mount (this will take a few minutes). Having a small number of OI files will cause scalability/performance issues.</p> <p>Mahmoud</p> <p>I'm just checking in on this one. Presumably you have the  <font color="#172b4d"><a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a> </font>fix in place on the NASA distribution now (and it is already merged for the upcoming 2.15.3 release) but has the restoration of the OI files for the impacted system been completed?</p> <p>Peter</p> <p>Closing this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-16655" title="Files not accessible after 2.12 -&gt; 2.14/2.15 upgrade" class="issue-link" data-issue-key="LU-16655"><del>LU-16655</del></a></p> Related LU-16655 Development Rank 1|i03hpb: Rank (Obsolete) 9223372036854775807 Severity