https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13241 Lustre <p>Running strace of <tt>mount.lustre</tt> shows an interesting reason why it spent a huge mount of time on a large filesystem. <tt>mount.lustre</tt> invokes several 'debugfs' command internally, but they took a huge amount of time here. Here is example.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10056 10:35:21.590086 execve("/sbin/debugfs", ["debugfs", "-c", "-R", "stat CONFIGS/mountdata", "/dev/sda"] 10056 10:39:00.257586 +++ exited with 0 +++ 10134 10:39:00.259343 execve("/bin/sh", ["sh", "-c", "debugfs -c -R 'dump /CONFIGS/mou"...] 10134 10:42:39.116052 +++ exited with 0 +++ </pre> </div></div> <p>Total mount time was 982 sec in this time. But, It took 419 sec for just those two debugfs calls against 982 sec.</p> <p>These two calls are just reading the <tt>CONFIGS/mountdata</tt> file from the filesystem, but that could be done directly via libext2fs, along with many of the other operations in <tt>libmount_utuls_ldiskfs.c</tt>, rather than launching external binaries to do the work. We should take care not to load the whole filesystem metadata, if that can be avoided. </p> LU-13241

mount.lustre for large filesystem runs slow debugfs commands

Improvement Minor Resolved Fixed Dongyang Li Andreas Dilger Tue, 11 Feb 2020 18:46:21 +0000 Mon, 31 Jan 2022 05:58:11 +0000 Wed, 18 Mar 2020 11:27:51 +0000 Lustre 2.14.0 0 6 <p>It <em>may</em> be that commit <tt>v1.45.4-16-ge6069a05</tt> "<tt>Teach ext2fs_open2() to honor the EXT2_FLAG_SUPER_ONLY flag</tt>" in the upstream e2fsprogs could improve the performance of <tt>debugfs</tt> in this case, since it will avoid reading the group descriptors in this case.</p> <p>That commit will reduce the time for tune2fs, and dumpe2fs -h, so we should see improvement when enabling the quota and mmp features.</p> <p>debugfs doesn't use that flag and if we are reading/writing the mountdata, we need to read the group descriptors anyway.</p> <p>I'm working on making libmount_utils_ldiskfs using the libext2fs, but we still need to do ext2fs_open() like you mentioned. Need to test if there's any improvement after that.</p> <p>We shouldn't need to read the group descriptors or bitmaps just to <em>read</em> the mountdata file, only when writing it (usually only right after formatting, when most of the filesystem is empty).</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37656" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37656</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13241" title="mount.lustre for large filesystem runs slow debugfs commands" class="issue-link" data-issue-key="LU-13241"><del>LU-13241</del></a> utils: use libext2fs for ldiskfs operations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bd12ea6a75e185f7d7c457164a0e0c528c26a824</p> <p>Looks like libext2fs doesn't handle quota, so if we use libext2fs to write the ldd, e2fsck will complain about inconsistent usage and quota accounting on the target, so I left the write part out.</p> <p>we also need to install e2fsprogs-devel on the build box from now on, I'm not sure if Jenkins is setup that way.</p> <p> </p> <p>Running e2fsck definitely handles file quota, but maybe only at the end. There is a quota library internal to e2fsprogs, but it isn't included as part of the -devel install.</p> <p>correct, I noticed there's a quota library but it's internal to e2fsprogs. I actually got the ldiskfs_write_ldd working with libext2fs as well, with all the lookup/allocate inode/link parent/write inode dance. However e2fsck complains the quota with the just created fs. maybe we can still write the ldd with libext2fs and use e2fsck to fix the quota at the end of mkfs.lustre?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37656/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37656/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13241" title="mount.lustre for large filesystem runs slow debugfs commands" class="issue-link" data-issue-key="LU-13241"><del>LU-13241</del></a> utils: use libext2fs for ldiskfs operations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7dc8aa7e7848f800c54eb18ecd59d665484ce822</p> <p>Although patch was alraedy landed, here is test resutls without/with patch <a href="https://review.whamcloud.com/37656/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37656/</a></p> <p>1 x 1.2PB OST and 'fake' filled up 50% fileystem by patch <a href="https://review.whamcloud.com/#/c/37329/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/37329/</a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Average mount time (Tried 3 times) Without patch: 72.3 sec (77s,70s,70s) With patch: 48.6 sec (51s,48s,47s) </pre> </div></div> <p>So, patch contributed 33% speedup of mount time.</p> <p>Landed for 2.14</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38027" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38027</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13241" title="mount.lustre for large filesystem runs slow debugfs commands" class="issue-link" data-issue-key="LU-13241"><del>LU-13241</del></a> libext2fs: add quota to libext2fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre-test<br/> Current Patch Set: 1<br/> Commit: 38122fbe0d40670efbf787141a549875b4186d4f</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38028" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38028</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13241" title="mount.lustre for large filesystem runs slow debugfs commands" class="issue-link" data-issue-key="LU-13241"><del>LU-13241</del></a> utils: covert to libext2fs for rest of ldiskfs ops<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d1a62adc3449fe9b16c0adc22f77e99393698afe</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38027/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38027/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13241" title="mount.lustre for large filesystem runs slow debugfs commands" class="issue-link" data-issue-key="LU-13241"><del>LU-13241</del></a> libext2fs: add quota to libext2fs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre-test<br/> Current Patch Set: <br/> Commit: ab3ebc4782c04ae25568a17e456c29441a0ce044</p> Related LU-1083 Development Rank 1|i00tmf: Rank (Obsolete) 9223372036854775807