https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8787 Lustre <p>On a DNE file system, MDT0000 ran out of space while one or more other MDTs were in recovery.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2016-10-31 18:26:53 [20537.964631] Lustre: Skipped 1 previous similar message 2016-10-31 18:26:58 [20542.793836] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) lsh-MDT0000: failed to start transaction due to ENOSPC. Metadata overhead is underestimated or grant_ratio is too low. 2016-10-31 18:26:58 [20542.815473] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) Skipped 39 previous similar messages 2016-10-31 18:26:58 [20542.827434] LustreError: 31561:0:(llog_cat.c:744:llog_cat_cancel_records()) lsh-OST0009-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -28 2016-10-31 18:26:58 [20542.843771] LustreError: 31561:0:(osp_sync.c:1031:osp_sync_process_committed()) lsh-OST0009-osc-MDT0000: can't cancel record: -28 </pre> </div></div> <p>Obviously the first step is to increase the capacity of the pool. However, after that is done, is further action required? Should I run lfsck, or do anything else?</p> Lustre: Build Version: 2.8.0_5.chaos LU-8787

zpool containing MDT0000 out of space

Question/Request Minor Closed Done nasf Olaf Faaland llnl Tue, 1 Nov 2016 23:43:22 +0000 Thu, 2 Nov 2017 22:02:52 +0000 Thu, 2 Nov 2017 22:02:52 +0000 0 4 <p>Fan Yong</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>I find that update_log_dir is taking 1.1T out of the 1.4T available. Shall I create a separate ticket for that? It seems far too large to me, but maybe I'm wrong.</p> <p>There are 158 files in update_log_dir.<br/> 68 size&gt;10GB<br/> 29 10GB &gt; size &gt;= 1GB<br/> 7 1GB &gt; size &gt;= 1M<br/> 44 size &lt; 1M</p> <p>Created a separate ticket <a href="https://jira.whamcloud.com/browse/LU-8794" title="update_log_dir consuming 1.1TB on MDT0000" class="issue-link" data-issue-key="LU-8794"><del>LU-8794</del></a> for the large amount of space occupied by update_log_dir. </p> <p>This ticket is only for the procedure to be followed when an MDT fills up, since it could happen in production and we need to know the procedure for recovering.</p> <p>thanks,<br/> Olaf</p> <p>According to current DNE implementation, the cross-MDTs operations (in detail) will be recorded as llog under the update_log_dir for recovery purpose. For most of use cases, the llog is append only, if there are too much cross-MDTs operations, the llog will become huge. So if you can describe your operations before the out of space, that may help us to judge the issue. Anyway, if there are some Lustre kernel debug logs on the MDT0000, that will be better.</p> <p>you can delete update_log* manually as what we did on <a href="https://jira.whamcloud.com/browse/LU-8753" title="Recovery already passed deadline with DNE" class="issue-link" data-issue-key="LU-8753"><del>LU-8753</del></a>, which I need further log.<br/> And there is also another ticket <a href="https://jira.whamcloud.com/browse/LU-8714" title="too many update logs during soak-test." class="issue-link" data-issue-key="LU-8714">LU-8714</a> about deleting update_log efficiently. I will try to work out the patch.</p> <p>Di,<br/> Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.<br/> thanks,<br/> Olaf</p> <p>nasf,<br/> I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.</p> <blockquote> <p>Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.<br/> ...<br/> I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.</p></blockquote> <p>Generally, the llog became bigger after the reboot means there are something to be recovered. But as long as your recovery complete successfully after the reboot, even if you removed the llogs, your namespace should be in consistent status unless there were some inconsistency before your reboot (for ZFS backend, it should very rare case).</p> <p>Olaf: <br/> it might be affected, but the chance is normally low as nasf said. Anyway I would suggest you to run lfsck to check and fix the consistency after you delete the update_logs.</p> <p>Olaf,<br/> Have you tried to remove the huge llogs as Wangdi suggested? Any feedback for that?<br/> Thanks</p> <p>Basic advice that we should delete the update logs and then run lfsck is a sufficient answer.</p> <p>This occurred during DNE2 testing with Lustre 2.8, which we have decided not to work at any further.  Instead we will test DNE2 when we start testing Lustre 2.10.x.  So we will test the advice only if we encounter the problem again, and in that case will file a new ticket.</p> Related LU-8753 Development Rank 1|hzyu3r: Rank (Obsolete) 9223372036854775807