https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7611 Lustre <p>We had a hard power outage early this morning. After hardware fixes, we were about to mount one of our file systems and it appears to be healthy.</p> <p>On the other file system, we have 5 OSTs that have reported unhealthy. The file system mounted fine, and clients connected, but soon after, I started receiving alerts. I'm in the process of rebooting the nodes so that I can e2fsck them. On each node that has an unhealthy OST, they have messages similar to the following:</p> <p>[ 2962.001970] LustreError: 24264:0:(tgt_lastrcvd.c:583:tgt_client_new()) atlas2-OST00ca: Failed to write client lcd at idx 19, rc -30<br/> [ 2962.029196] LustreError: 24264:0:(tgt_lastrcvd.c:583:tgt_client_new()) Skipped 140 previous similar messages<br/> [ 3263.951324] LustreError: 24172:0:(ofd_obd.c:1365:ofd_create()) atlas2-OST00ca: unable to precreate: rc = -30<br/> [ 3263.981018] LustreError: 24172:0:(ofd_obd.c:1365:ofd_create()) Skipped 61 previous similar messages<br/> [ 3562.220443] LustreError: 24251:0:(tgt_lastrcvd.c:583:tgt_client_new()) atlas2-OST00ca: Failed to write client lcd at idx 19, rc -30<br/> [ 3562.252970] LustreError: 24251:0:(tgt_lastrcvd.c:583:tgt_client_new()) Skipped 233 previous similar messages<br/> [ 3874.190928] LustreError: 24281:0:(ofd_obd.c:1365:ofd_create()) atlas2-OST00ca: unable to precreate: rc = -30<br/> [ 3874.219280] LustreError: 24281:0:(ofd_obd.c:1365:ofd_create()) Skipped 62 previous similar messages</p> <p>Is an e2fsck the proper response in this type of situation?</p> <p>Thanks,<br/> &#8211;<br/> Jesse</p> RHEL 6.6<br/> 2.6.32-504.30.3.el6.atlas.x86_64 LU-7611

OSTs become "not healthy"

Bug Major Resolved Done Jian Yu Jesse Hanley Thu, 24 Dec 2015 21:45:16 +0000 Mon, 28 Mar 2016 15:08:02 +0000 Mon, 28 Mar 2016 15:08:02 +0000 Lustre 2.5.3 0 7 <p>Yes, Jesse. Please refer to <a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.html#dbdoclet.50438225_71141" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.html#dbdoclet.50438225_71141</a> .</p> <p>Thanks Jian! About to start the e2fsck runs now.</p> <p>Thanks Jesse. I'm asking Jian to watch this ticket, for the time being.</p> <p>Please let us know how the e2fsck runs proceed for you.</p> <p>~ jfc.</p> <p>Sorry for the delay on this - we managed to get the file systems back up. We ran e2fscks on all target OSTs during an outage and got the following output (I removed some of the superfluous and <tt><span class="error">&#91;QUOTA WARNING&#93;</span></tt> lines):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">atlas1-OST02e8: 1471186/29343744 files (7.2% non-contiguous), 2281519450/3755999232 blocks Inode 5712358, i_size is 2097152, should be 2138112. Fix? no atlas1-OST00aa: 1418001/29343744 files (7.1% non-contiguous), 2267986225/3755999232 blocks Deleted inode 1184018 has zero dtime. Fix? no Inode bitmap differences: -1184018 Fix? no atlas1-OST0212: 1530414/29343744 files (7.2% non-contiguous), 2274401960/3755999232 blocks Deleted inode 94387 has zero dtime. Fix? no Inode bitmap differences: -94387 Fix? no atlas1-OST0088: 1246831/29343744 files (7.4% non-contiguous), 2518723619/3755999232 blocks [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (1527054912 &gt;= 2690) in user quota file. Quota file is probably corrupted. Please run e2fsck (8) to fix it. [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (20480 &gt;= 2690) in user quota file. Quota file is probably corrupted. Please run e2fsck (8) to fix it. [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (897567104 &gt;= 2690) in user quota file. Quota file is probably corrupted. Please run e2fsck (8) to fix it. [ERROR] quotaio_tree.c:590:check_reference:: Illegal reference (4096 &gt;= 2690) in user quota file. Quota file is probably corrupted. Please run e2fsck (8) to fix it. atlas2-OST033f: 1181549/29343744 files (1.5% non-contiguous), 2164774412/3755999232 blocks Deleted inode 1791 has zero dtime. Fix? no Block bitmap differences: -(77247232--77247743) Fix? no Inode bitmap differences: -1791 Fix? no atlas2-OST0228: 1277960/29343744 files (1.6% non-contiguous), 1921932726/3755999232 blocks Deleted inode 4916172 has zero dtime. Fix? no Inode bitmap differences: -4916172 Fix? no </pre> </div></div> <p>Do you see anything alarming about this output, or should we be fine to run e2fsck (with -y or -p).</p> <p>Thanks,<br/> &#8211;<br/> Jesse</p> <p>Hi Niu,</p> <p>Could you please take a look at the above outputs of e2fsck and advise? Thank you.</p> <p>I think it should be fine to run e2fsck -y to repair the system, please make sure to use the latest recommended e2fsprogs (1.42.13.wc4, I believe), which has several recent defect fixing included.</p> <p>Hello Jesse,</p> <p>Do you need any more work done on this ticket?</p> <p>Many thanks,<br/> ~ jfc.</p> <p>Hey John,</p> <p>We were able to complete the e2fsck runs without an issue. Everything appeared to be in a healthy state afterwards. Please feel free to resolve this, and thanks for the help and advice.</p> <p>&#8211;<br/> Jesse</p> <p>Thank you Jesse &#8211; glad everything worked out well.</p> <p>~ jfc.</p> Development Rank 1|hzxwtj: Rank (Obsolete) 9223372036854775807 Severity