https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1569 Lustre <p>We have had a catastrophic failure of one of our lustre filesystems. Not sure exactly cause, but in our current state running lfsck on it gives TONS of errors like:<br/> Failed to find fid <span class="error">&#91;0xc900ec:0xd7e33ef4:0x0&#93;</span>: DB_NOTFOUND: No matching key/data pair found</p> <p>And when we run a find on various users' directories, we find many "No such file" errors:<br/> find: ./mrtoeppe/CFD_run_archive/Turbine2DecDet/mcfd_tec.bin.830: No such file or directory which shows up like:</p> <p>?--------- ? ? ? ? ? mcfd_tec.bin.660<br/> ?--------- ? ? ? ? ? mcfd_tec.bin.670<br/> ?--------- ? ? ? ? ? mcfd_tec.bin.680</p> CentOS release 5.7 (Final)<br/> Linux nas-0-1.local 2.6.18-194.17.1.el5_lustre.1.8.5 #1 SMP Tue Nov 16 17:59:07 MST 2010 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-1569

Many Files missing and others have no info (uid/gid/permissions)

Task Critical Resolved Incomplete WC Triage Brian Andrus Tue, 26 Jun 2012 14:42:02 +0000 Wed, 6 Nov 2013 17:40:29 +0000 Wed, 6 Nov 2013 17:40:29 +0000 Lustre 1.8.7 0 2 <p>Have you successfully run 'fsck -fy' on all devices? Are you using the latest version of e2fsprogs, available at <a href="http://downloads.whamcloud.com/public/e2fsprogs/" class="external-link" target="_blank" rel="nofollow noopener">http://downloads.whamcloud.com/public/e2fsprogs/</a></p> <p>Initially our lustre filesystem (/work) had one of the osts disconnect (there are 10 each 7.8TB OSTs) and not reconnect. This put /work in read-only mode.<br/> I attempted to reconnect work_ost9, but it failed with Transport Endpoint shutdown errors (odd for an OST I think).<br/> I took the entire system down and ran fsck on each ost and the mdt. There were numerous errors on work_ost9 as well as a few errors on the mdt and 2 other OSTs.<br/> Upon remount, we found that there were almost no files newer than December 12, 2011.<br/> We also found many files were corrupt and did not show proper UID/GID/Permissions.<br/> There WERE files accessed and written to by users during this time.<br/> It seemed there may have been issues with the LAST_ID and/or CATALOG on the MDT, so those were removed and the system brought back online. The made the LAST_ID entries on the MDT match those listed on the OSTs.<br/> Upon remounting (read only now), we found the corrupt entries were still there and there were still no files newer than December.<br/> I took down the filesystem again and ran e2fsck to create the MDT and OST databases. I then brought the filesystem back up and ran lfsck in read only.<br/> This produced many of the "No matching key/data pair found" errors.<br/> I ran lfsck without "-n" with the same result.</p> <p>Currently /work is mounted read only so users that do still have data intact can copy it to a clean filesystem.</p> <p>Okay, thanks </p> <p>First, as explained in the Lustre Manual, lustre-logs which are auto-dumped must be pre-processed on site to be useful, so we can't do much with what you attached. What we need in this case are the <em>system</em> logs (typically /var/log/messages) for all OSTs and the MDS/MGS for the period 12 hrs before you had the initial outage to the present time. Please do not filter the logs unless you need to remove IPs for security. </p> <p>Please run lfs getstripe on one of the missing files, get the list of stripe objects and check the OSTs to determine if the data actually exists on the OST disk. Debugfs will work for this. <br/> The lfs setstripe should return a list of ostidx (OST index) and object IDs, for an objid of (example) 818855<br/> the following debugfs command should tell you if the data is there. </p> <p>$ debugfs -c -R "stat O/0/d$((818855 % 32))/818855" /dev/&lt;your OST device&gt; </p> <p>Here is a quick check on one file that is showing in an ls, but missing info:</p> <p><span class="error">&#91;root@nas-0-1 hale&#93;</span># ls -l|grep gempak$<br/> ?--------- ? ? ? ? ? gempak<br/> <span class="error">&#91;root@nas-0-1 hale&#93;</span># lfs getstripe ./gempak<br/> ./gempak<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_stripe_offset: 0<br/> obdidx objid objid group<br/> 0 5703559 0x570787 0</p> <p><span class="error">&#91;root@nas-0-1 hale&#93;</span># debugfs -c -R "stat O/0/d$((5703559 % 32))/5703559" /dev/VG_hamming/work_ost0<br/> debugfs 1.41.12.2.ora1 (14-Aug-2010)<br/> /dev/VG_hamming/work_ost0: catastrophic mode - not reading inode or group bitmaps<br/> O/0/d7/5703559: File not found by ext2_lookup</p> <p>Tar file of /var/log/messages for MGS and OSSes</p> <p>Thanks - did you keep any logs/output from the first fsck you did after the initial failure? Please attach if so.</p> <p>The only log I have is the output from lfsck, but it is 7.9GB<br/> I can gzip it and try to upload if you think it will help.</p> <p>Attached output from running lctl df on all the dump logs that were generated (lustre.log)</p> <p>We need the fsck data, not the lfsck.</p> <p>That I do not have. I do know there are many files in lost+found on the backing filesystem. I have not examined them yet since it is now mounted as lustre (albeit read-only).</p> <p>Have you run the lost+found recovery script?</p> <p>That would be "ll_recover_lost_found_objs", which should be installed on all the OSTs. You need to mount the OST locally using "-t ldiskfs" instead of as "-t lustre" to run this tool. It will rebuild the corrupted object directories and move all the objects from lost+found back into their proper location.</p> Development Epic server Rank 1|hzv33r: Rank (Obsolete) 4002