https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9196 Lustre <p>Our MDS server crashed due to memory exhaustion. Examination of the system logs show nothing out of the ordinary expect it was noticed that an IO scrub did start off on the MDS server:</p> <p>Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the <span class="error">&#91;0x20003f37b:0x3e5:0x0&#93;</span> with flags 0x4a, rc = 0</p> <p>Some time after that we encountered the following crash which is attached.</p> <p> </p> RHEL6.8 running non patched Lustre 2.8 server using ldiskfs. LU-9196

MDS server for Atlas file system crashed due to memory exhaustion.

Bug Minor Resolved Fixed nasf James A Simmons Wed, 8 Mar 2017 19:01:08 +0000 Tue, 5 Jun 2018 16:36:51 +0000 Tue, 5 Jun 2018 16:36:51 +0000 Lustre 2.8.0 0 3 <p>The IO scrubber was triggered at 02:06, 05:12, 08:26, 11:36, 14:47 during the day.</p> <p>Hi Nasf,<br/> Could you please advise? Thank you.</p> <p>According to the logs, there are several issues:</p> <p>1) OI scrub was triggered because of OI inconsistency, including the following three FIDs:<br/> <span class="error">&#91;0x1000:0x15c5020:0x0&#93;</span><br/> <span class="error">&#91;0x20003f37b:0x3e5:0x0&#93;</span><br/> <span class="error">&#91;0x20003f47a:0xd8e0:0x0&#93;</span></p> <p>Would you please to find out the file or object/inode corresponding to these FIDs via "lfs fid2path". The FID <span class="error">&#91;0x1000:0x15c5020:0x0&#93;</span> is IGIF, some abnormal. Please dump such inode (#4096) via debugfs.</p> <p>2) LBUG() during osd_object_release().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;[2339122.255892] LustreError: 16352:0:(osd_handler.c:1610:osd_object_release()) LBUG &lt;4&gt;[2339122.264739] Pid: 16352, comm: mdt01_381 &lt;4&gt;[2339122.269446] &lt;4&gt;[2339122.269446] Call Trace: &lt;4&gt;[2339122.274687] [&lt;ffffffffa05b4875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[2339122.282893] [&lt;ffffffffa05b4e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[2339122.290225] [&lt;ffffffffa0ea93e8&gt;] osd_object_release+0x88/0x90 [osd_ldiskfs] &lt;4&gt;[2339122.298765] [&lt;ffffffffa074d6fd&gt;] lu_object_put+0x16d/0x3b0 [obdclass] &lt;4&gt;[2339122.306500] [&lt;ffffffffa102abc7&gt;] mdt_getattr_name_lock+0x5f7/0x1900 [mdt] &lt;4&gt;[2339122.314609] [&lt;ffffffffa102c3f2&gt;] mdt_intent_getattr+0x292/0x470 [mdt] &lt;4&gt;[2339122.322330] [&lt;ffffffffa101d93e&gt;] mdt_intent_policy+0x4be/0xc70 [mdt] &lt;4&gt;[2339122.329981] [&lt;ffffffffa091c0c7&gt;] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] &lt;4&gt;[2339122.337912] [&lt;ffffffffa0946307&gt;] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] &lt;4&gt;[2339122.346565] [&lt;ffffffffa09b9a71&gt;] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] &lt;4&gt;[2339122.354501] [&lt;ffffffffa09cbbe1&gt;] tgt_enqueue+0x61/0x230 [ptlrpc] &lt;4&gt;[2339122.361753] [&lt;ffffffffa09cc69c&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] &lt;4&gt;[2339122.369877] [&lt;ffffffffa09796f1&gt;] ptlrpc_main+0xd21/0x1800 [ptlrpc] &lt;4&gt;[2339122.377324] [&lt;ffffffffa09789d0&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] &lt;4&gt;[2339122.384747] [&lt;ffffffff810a640e&gt;] kthread+0x9e/0xc0 &lt;4&gt;[2339122.390625] [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 &lt;4&gt;[2339122.396587] [&lt;ffffffff810a6370&gt;] ? kthread+0x0/0xc0 &lt;4&gt;[2339122.402556] [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>It seems that the inode nlink attribute is invalid (be marked as zero but nobody destroy it). We hit similar trouble in <a href="https://jira.whamcloud.com/browse/LU-8992" title="osd_object_release() LBUG" class="issue-link" data-issue-key="LU-8992"><del>LU-8992</del></a>. But only with the given logs, I cannot say they are the same.</p> <p>3) A lot of mdt_getattr_internal() failure as following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;[2230069.055437] LustreError: 16356:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003863b4:0xc7fd:0x0]: rc = -2 </pre> </div></div> <p>It may be normal because of raced unlink operation from other. Let's ignore this failure temporarily.</p> <p>4) Some directories are full:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;[401753.420890] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 &lt;4&gt;[401753.434706] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem </pre> </div></div> <p>It is because ldiskfs only supports two levels htree-based directory by default. If too many entries are inserted into single directory, it will be exhausted although there are still space on disk.</p> <p>5) Out of memory.<br/> Currently, I am not sure who exhausted the RAM, the first suspected point is NOT OI scrub, instead, it may be related with the possible improperly cleanup the page-cache after directory full.</p> <p>Any further feedback or logs or reproduction?</p> <p>We haven't seen this problem since. As for the OI problems you saw we are running lfsck to clean those up. Once lfsck is done I will report if everything is fixed.</p> <p>Any update? Thanks!</p> <p>Hi James,</p> <blockquote><p>We haven't seen this problem since. As for the OI problems you saw we are running lfsck to clean those up. Once lfsck is done I will report if everything is fixed.</p></blockquote> <p>Is everything fixed after running lfsck?</p> <p>The main issues should have been fixed via LFSCK. Please reopen it if have more questions.</p> Development Rank 1|hzz6bz: Rank (Obsolete) 9223372036854775807 Severity