https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8794 Lustre <p>On zinc, a DNE filesystem with 16 MDTs, the pool containing MDT0000 (zinc1) ran out of space. Upon inspection, we find that 1.1 TB is occupied by files contained in updat_log_dir. The rest of the MDT occupies about 300MB, which is about the same as the space used by each of the other 15 MDTs.</p> Lustre: Build Version: 2.8.0_5.chaos LU-8794

update_log_dir consuming 1.1TB on MDT0000

Bug Critical Resolved Cannot Reproduce Lai Siyao Olaf Faaland llnl Wed, 2 Nov 2016 21:19:38 +0000 Wed, 29 Nov 2017 19:00:14 +0000 Wed, 29 Nov 2017 19:00:14 +0000 0 4 <p>There are 158 files in update_log_dir.<br/> 68 size&gt;10GB<br/> 29 10GB &gt; size &gt;= 1GB<br/> 7 1GB &gt; size &gt;= 1M<br/> 44 size &lt; 1M</p> <p>Hi Lai,</p> <p>Can you please take a look at this issue?</p> <p>Thanks.<br/> Joe</p> <p>Unfortunately I cannot be certain of the filesystem activity that caused this. We were not monitoring the space usage in the pool (although we are now).</p> <p>I also cannot provide debug logs from the MDTs, as we discovered the problem after a reboot of the servers.</p> <p>The only information available is syslog output for the servers and the contents of the MDT itself.</p> <p>Di Wang suggested I can delete the contents of update_log_dir. Let me know if you need any information about its contents before I do that.</p> <p>Note that this ticket is purely for trying to figure out why the update logs are occupying so much space. There is a separate ticket, <a href="https://jira.whamcloud.com/browse/LU-8787" title="zpool containing MDT0000 out of space" class="issue-link" data-issue-key="LU-8787"><del>LU-8787</del></a>, for how to recover.</p> <p>If the contents of the MDT won't help us learn what happened, we can just close the ticket until it happens again and we can get better information.<br/> We have started monitoring space used in the pool containing the MDT, and will be more likely to notice if the volume of update logs increases.</p> <p><a href="http://review.whamcloud.com/18028" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18028</a> (<a href="https://jira.whamcloud.com/browse/LU-6838" title="update llog become too big before it is destroyed" class="issue-link" data-issue-key="LU-6838"><del>LU-6838</del></a>) might help here, but as it explained there, the plain log limit size is around 800M, probably can not explain why the update log file reach to 1T. something is strange here. anyway I think the suggestion on <a href="https://jira.whamcloud.com/browse/LU-8714" title="too many update logs during soak-test." class="issue-link" data-issue-key="LU-8714">LU-8714</a> is the way to go.</p> <p>I was unable to reproduce the problem after it was initially encountered, and we have not seen it since on test or production systems since then, perhaps because we have not been testing DNE2 and use very few remote directories. Closing.</p> Related LU-8714 LU-6838 Development Rank 1|hzyu9j: Rank (Obsolete) 9223372036854775807 Severity