https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8856 Lustre <p>End Customer: MSU (Michigan State Univ)</p> <p>A user generated tons of small files and exhausted the available inodes of the MDT (single MDT, no DNE). Any attempts at deleting files as root fail. </p> <p>I looked at <a href="https://jira.whamcloud.com/browse/LU-8787" title="zpool containing MDT0000 out of space" class="issue-link" data-issue-key="LU-8787"><del>LU-8787</del></a> and <a href="https://jira.whamcloud.com/browse/LU-8714" title="too many update logs during soak-test." class="issue-link" data-issue-key="LU-8714">LU-8714</a> but they don't seem to follow this closely enough.</p> <p>zdb -d ls15-mds-00.mdt/mdt<br/> Dataset ls15-mds-00.mdt/mdt <span class="error">&#91;ZPL&#93;</span>, ID 66, cr_txg 20442, 2.82T, 280362968 objects</p> <p>ls15-mds-00.mdt/mdt 2.82T 0 2.82T /ls15-mds-00.mdt/mdt</p> <p><span class="error">&#91;root@lac-373 roth&#93;</span># lfs df -i<br/> UUID Inodes IUsed IFree IUse% Mounted on<br/> ls15-MDT0000_UUID 280362968 280362968 0 100% /mnt/ls15<span class="error">&#91;MDT:0&#93;</span></p> <p>But we can't remove any files:</p> <p><span class="error">&#91;root@lac-000 1mk5_5998&#93;</span># rm tor.mat<br/> rm: cannot remove `tor.mat': No space left on device</p> <p>I'm going to take a stab at deregistering the changelog which might free up enough space to get the MDT able to process some file deletions. If anyone has any other 'best practices' please advise.</p> CentOS 6.8 2.6.32_504.30.3.el6.x86_64, Lustre 2.8.0 (g0bcd520), ZFS 0.6.5.4-1 LU-8856

ZFS-MDT 100% full. Cannot delete files.

Bug Critical Resolved Fixed Alex Zhuravlev Jeff Johnson llnl Mon, 21 Nov 2016 18:43:32 +0000 Tue, 6 Feb 2024 06:44:07 +0000 Thu, 15 Mar 2018 14:04:43 +0000 Lustre 2.8.0 Lustre 2.11.0 Lustre 2.10.4 0 8 <p>If you have ChangeLogs active without an active consumer, then this will definitely consume a lot of space that does not get freed until the ChangeLog is processed or removed. Also, having an active ChangeLog means that some space is needed for the CL record at unlink time.</p> <p>Do you have any snapshots of this filesystem? If yes, then deleting the oldest snapshot should also free up some space.</p> <p>It may be that mounting and unmounting the dataset (up to 4 times) will allow old committed transactions to free up space</p> <p>If none of these options work, it may be possible to mount the filesystem locally as type <tt>zfs</tt> and deleting some specific files, however we should discuss that before any action is taken like this.</p> <p>Finally, one option would be to add extra storage to the MDT zpool. However, note that it will <b>not</b> be possible to remove those devices after they are added, so if this is done they should be configured correctly as mirrored VDEV(s) to maintain reliability.</p> <p>Draining changelogs or deregistering the changelog isn't working. For some reason the changelog doesn't have a user. The user <b>was</b> cl1 and the logs of the robinhood server show it was processing using cl1.</p> <p>On MDS it appears that there are no unprocessed changelog entries but robinhood was running up until a few months ago so there should be unprocessed changes stored:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /proc/fs/lustre/mdd/ls15-MDT0000/changelog_users current index: 164447373 ID index </pre> </div></div> <p>From a client, as root: (produces lots of output)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs changelog ls15-MDT0000|head 160211907 12LYOUT 01:50:56.131136452 2015.11.01 0x0 t=[0x200002cf8:0x12e9f:0x0] 160211908 12LYOUT 01:50:56.131136452 2015.11.01 0x0 t=[0x200002d22:0x167e9:0x0] 160211909 13TRUNC 01:50:56.132136455 2015.11.01 0xe t=[0x200002ca8:0x8a69:0x0] 160211910 13TRUNC 01:50:56.132136455 2015.11.01 0xe t=[0x200002cb4:0x17a3d:0x0] 160211911 11CLOSE 01:50:56.132136455 2015.11.01 0x42 t=[0x200002c37:0x8577:0x0] 160211912 11CLOSE 01:50:56.133136458 2015.11.01 0x42 t=[0x200002ca8:0x8a69:0x0] 160211913 11CLOSE 01:50:56.133136458 2015.11.01 0x42 t=[0x200002cb4:0x17a3d:0x0] </pre> </div></div> <p>Trying to clear as root from a client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs changelog_clear ls15-MDT0000 cl1 0 changelog_clear error: No such file or directory </pre> </div></div> <p>Trying to deregister from the MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ls15-mds-00.i ~]# lctl --device ls15-MDT0000 changelog_deregister cl1 error: changelog_deregister: No such file or directory [root@ls15-mds-00.i ~]# lctl --device ls15-MDT0000 changelog_deregister cl0 error: changelog_deregister: expected id of the form cl&lt;num&gt; got 'cl0' deregister an existing changelog user usage: device &lt;mdtname&gt; changelog_deregister &lt;id&gt; run &lt;command&gt; after connecting to device &lt;devno&gt; --device &lt;devno&gt; &lt;command [args ...]&gt; </pre> </div></div> <p>Logs from robinhood server showing consumption of changelogs using reader_id 'cl1':</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>======== General statistics ========= Daemon start time: 2016/07/28 18:48:59 Started modules: log_reader ChangeLog reader #0: fs_name = ls15 mdt_name = MDT0000 reader_id = cl1 records read = 4235467 interesting records = 2823646 suppressed records = 1411821 records pending = 0 last received = 2016/07/28 19:29:26 last read record time = 2015/10/31 22:28:52.489794 last read record id = 164447373 last pushed record id = 164447370 last committed record id = 164447370 last cleared record id = 164447370 read speed = 0.00 record/sec (0.00 incl. idle time) processing speed ratio = 0.00 ChangeLog stats: MARK: 0, CREAT: 0, MKDIR: 0, HLINK: 0, SLINK: 0, MKNOD: 0, UNLNK: 0, RMDIR: 0, RENME: 0 RNMTO: 0, OPEN: 0, CLOSE: 1411823, LYOUT: 1411822, TRUNC: 1411822, SATTR: 0, XATTR: 0 HSM: 0, MTIME: 0, CTIME: 0, ATIME: 0 </pre> </div></div> <p>There are no snapshots in the MDT pool.</p> <p>I was hoping to figure out how to see the changelog file or directory using zdb but I can't seem to find which object ID it might be. With over 2T full there are lots of entries to try and poke at. Is there a any sort of default object ID for the changelog file(s) or directory?</p> <p>By 'dataset' are you referring to unmounting and remounting the LFS server-side targets? Basically take down and remount the server-side of the LFS 3-4 times?</p> <p>We threw hardware at it. Expanded the MDT pool by adding a mirrored vdev and the extra 320GB gave room to move around and delete files. </p> <p>I'd still like to walk down this ticket so some best practices could be offered in the event a future occurrence doesn't have extra hardware at hand.</p> <p>While we do try to reserve space in the MDT and OST zpools (<tt>OSD_STATFS_RESERVED_SIZE</tt>), but I suspect we are not taking this into account when allocating files on the MDT, only on the OST.</p> <p>Separately, we need to look into how ChangeLogs are handled when the MDT is "full". The "unused ChangeLog is filling MDT" problem seems to be happening a lot. I think we need to handle this in an automatic manner, by tracking how much space the ChangeLog consumes, and if the MDT is too full and the oldest ChangeLog user that hasn't been used in some time (a week?) should be unregistered (with a clear LCONSOLE() error message printed) and records purged up to the next CL user. CL deregistration should be repeated in LRU order as needed until enough free space is available or no more unused CL users exist. It shouldn't automatically deregister active CL users (e.g. less than one day) since that could be used as a DOS to deactivate filesystem monitoring tools.</p> <p>A /proc tunable should be available to <em>disable</em> automatic CL user deregistration, and when this is set users would get ENOSPC instead of success when trying to modify the MDT. This should not be the default behaviour, however, and only used if it is more important to track every filesystem operation than it is to be able to use the filesystem.</p> <p>Lai</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Two things need to be done here to handle this problem automatically, since this problem of ChangeLogs filling the MDT has happened several times:</p> <ol> <li>if the MDT is too full and the ChangeLog consumes too much space (see also patch <a href="https://review.whamcloud.com/16416" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/16416</a> "<a href="https://jira.whamcloud.com/browse/LU-7156" title="Provide size of changelogs" class="issue-link" data-issue-key="LU-7156"><del>LU-7156</del></a> mdd: add changelog_size to procfs"), and some ChangeLog user hasn't been used in some time (over a week?) it should be unregistered (with a clear <tt>LCONSOLE()</tt> error message printed) and records purged up to the next CL user (this should happen automatically), repeat for the next CL user as needed. Possibly a <tt>/sys/fs/lustre</tt> tunable to disable automatic CL user deregistration should be available, and further operations on the filesystem would get ENOSPC instead of success (as it does today), if it really is critical to track every operation, but that should not be the default behavior. The deregistration should not be done for recently active ChangeLog users (&lt; 24h), since this would potentially allow users to disable the ChangeLogs just by filling the MDT, and there is little benefit to removing the CL user if it does not free up much space.</li> <li>reserve more space for ZFS filesystems, and when the threshold is hit only allow files to be deleted. This needs to be done in conjunction with automatic removal of old ChangeLog, otherwise deleting files will free some space (assuming whole metadnode blocks are freed) but it will also consume space in the ChangeLog.</li> </ol> <p>One option for a very simple short-term solution for the ZFS space reservation is to have the MDS or OSD startup check the size of and/or write a 10MB <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> file in the MDS root with a name like <tt>IN_CASE_OF_ENOSPC_TRUNCATE_THIS_FILE</tt>.  This would be large enough to ensure that truncating the file manually from a local mount if there is an emergency situation like this in the future releases enough space to start deleting file again.  It isn't high-tech, but is definitely a robust way to reserve space for such an emergency, and in most cases that space won't be missed.  The size of the file could be scaled down for small test filesystems below, say, 10GB or skipped completely.  Some care would be needed to avoid refilling the file immediately after mount if the MDS is just being mounted after truncating the <tt>ICE</tt> file and files are being deleted. </p> <p>However, it shouldn't delay too long in repopulating the file to avoid the situation where there is some runaway user job that continues to fill the filesystem and it gets back into the same situation again immediately.  The benefit of this low-tech approach (vs. an in-memory reservation of space, and selectively blocking all but file/directory removal operations) is that this could be implemented quickly and potentially backported to existing releases with little risk.</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26868" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26868</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: reserve space in zfs pool for emergency<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5920a86e0a1fad6669997483a28fcea95d9a2fce</p> <p>Andreas, probably there is another solution for the problem.</p> <p>Basically ZFS reserves some space internally:</p> <ul> <li>Normally, we don't allow the last 3.2% (1/(2^spa_slop_shift)) of space in</li> <li>the pool to be consumed. This ensures that we don't run the pool</li> <li>completely out of space, due to unaccounted changes (e.g. to the MOS).</li> <li>It also limits the worst-case time to allocate space. If we have</li> <li>less than this amount of free space, most ZPL operations (e.g. write,</li> <li>create) will return ENOSPC.<br/> *</li> <li>Certain operations (e.g. file removal, most administrative actions) can</li> <li>use half the slop space. They will only return ENOSPC if less than half</li> <li>the slop space is free. Typically, once the pool has less than the slop</li> <li>space free, the user will use these operations to free up space in the pool.</li> <li>These are the operations that call dsl_pool_adjustedsize() with the netfree</li> <li>argument set to TRUE.</li> </ul> <p>we can mark any transction "net free" using dmu_tx_mark_netfree()</p> <p>so the very first thing would be to mark transactions involving object destroy.<br/> then we could have a procfs tunable so that sysadm can turn that for specificic transactions (e.g. originated from root).</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26930" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26930</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 63b56104f21e8e1abfe962bd9ab6b749b67fed3a</p> <p>the approach seem to work (in simple cases at least). here is the test:</p> <p>test_803() {<br/> mkdir $DIR/$tdir<br/> createmany -m $DIR/$tdir/f 10000000000 &amp;&amp; error "too big device?"<br/> rm -rf $DIR/$tdir || error "rm should succeed after ENOSPC"<br/> }</p> <p>== sanity test 803: OOS == 15:52:41 (1493902361)</p> <ul class="alternate" type="square"> <li>create 10000 (time 1493902365.56 total 4.41 last 2269.00)</li> <li>create 20000 (time 1493902371.51 total 10.35 last 1681.47)</li> <li>create 30000 (time 1493902377.66 total 16.50 last 1625.92)</li> <li>create 40000 (time 1493902384.95 total 23.80 last 1370.67)</li> <li>create 44433 (time 1493902394.95 total 33.80 last 443.18)</li> <li>create 46390 (time 1493902404.97 total 43.81 last 195.48)</li> <li>create 47995 (time 1493902414.97 total 53.82 last 160.40)</li> <li>create 49455 (time 1493902424.98 total 63.83 last 145.92)</li> <li>create 50000 (time 1493902428.78 total 67.62 last 143.49)</li> <li>create 51395 (time 1493902438.78 total 77.63 last 139.39)</li> <li>create 52925 (time 1493902448.79 total 87.64 last 152.94)</li> <li>create 54468 (time 1493902458.79 total 97.64 last 154.29)</li> <li>create 56076 (time 1493902468.80 total 107.64 last 160.70)</li> <li>create 57716 (time 1493902478.80 total 117.65 last 163.87)</li> <li>create 59290 (time 1493902488.81 total 127.66 last 157.27)</li> <li>create 60000 (time 1493902493.28 total 132.12 last 159.07)</li> <li>create 61487 (time 1493902503.28 total 142.13 last 148.58)<br/> mknod(/mnt/lustre/d803.sanity/f62098) error: No space left on device<br/> total: 62098 create in 146.27 seconds: 424.54 ops/second<br/> Resetting fail_loc on all nodes...done.<br/> 15:56:09 (1493902569) waiting for dual2 network 5 secs ...<br/> 15:56:09 (1493902569) network interface is UP<br/> PASS 803 (208s)</li> </ul> <p>w/o the patch rm fails..</p> <p>unfortunately this capability was added in 0.7, it's not easily available on 0.6 though the majority of the required functionality is in place.</p> <p>I think the two approaches are complimentary. We can use the reserved space file for now, and use the "netfree" functionality when it is available.</p> <p>The main question about "netfree" is whether this is actually true when we delete an inode on the MDT with ChangeLogs enabled? Even if the dnode is deleted, it may not actually release space (due to shared dnode blocks) and the added ChangeLog record will consume space. </p> <p>As a result, even if the netfree functionality is available I think it makes sense to keep the emergency space reservation file around. If we never need to delete it then that is fine too, the amount of space consumed is minimal. </p> <p>I think this is true for "reserved with writes" as well - changelogs/destroy logs can be quite big so that with that reserved released we'll keep consuming?<br/> correct me if I'm wrong, but I don't really see big difference.</p> <p>I think in the "reserved with writes" case, since the admin needs to get involved they can hopefully fix the source of the problem that is consuming all the free space (e.g. stale ChangeLog consumer registered) when they delete the emergency file.</p> <p>well, I guess we can mark any transaction originated from root with netfree flag when a special tunable set ?<br/> if no space can be released, admin comes in settting that variable and do whatever may help with it's rights..</p> <p>We've encountered this at LLNL, too.</p> <p>For the benefit of other sites that end up looking at this ticket and have Lustre versions without Alex's patches, I'm working up a procedure which I'll put on wiki.lustre.org at <a href="http://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/ZFS_MDT_ENOSPC_Recovery</a>. It will work on any ZFS &gt;= 0.6.5 using spa_slop_shift mentioned by Alex, above.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26930/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26930/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8d1639b5cf1edbc885876956dcd6189173c00955</p> <p>Landed for 2.11</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31442" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31442</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 83be173e2848e3b81d6fb2123d70d0cf614105a8</p> <p>Reopening due to <a href="https://jira.whamcloud.com/browse/LU-10732" title="sanity-lfsck test_9a: FAIL: (7) Failed to get expected &#39;completed&#39;" class="issue-link" data-issue-key="LU-10732"><del>LU-10732</del></a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31442/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31442/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c2caa40bd38e7645dc4ac90552e12e3fb7fde476</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31443" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31443</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d47e3892f83d2d1bdb21653381a8d1ec0db68a4a</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31444" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31444</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8bc89e5b3aeda1bf15f2ff6fc53651e470c0a6c6</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31444/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31444/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 106abc184d8b57de560dc1874683ce5487dcf30a</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31751" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31751</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: aaafe65b47925dd83b2138d55843ac61af2967a8</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31751/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31751/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8856" title="ZFS-MDT 100% full. Cannot delete files." class="issue-link" data-issue-key="LU-8856"><del>LU-8856</del></a> osd: mark specific transactions netfree<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: a8c7a32fd7fc54e9717e23f208b40c8ff93b81e4</p> Duplicate Related LU-7340 LU-10732 Development Epic metadata zfs Epic/Theme zfs Rank 1|hzyw7j: Rank (Obsolete) 9223372036854775807 Severity