[LU-13241] mount.lustre for large filesystem runs slow debugfs commands

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Labels: scalability
• Assignee: Dongyang Li
• Reporter: Andreas Dilger

Description

Running strace of mount.lustre shows an interesting reason why it spent a huge mount of time on a large filesystem. mount.lustre invokes several 'debugfs' command internally, but they took a huge amount of time here. Here is example.

10056 10:35:21.590086 execve("/sbin/debugfs", ["debugfs", "-c", "-R", "stat CONFIGS/mountdata", "/dev/sda"]
10056 10:39:00.257586 +++ exited with 0 +++

10134 10:39:00.259343 execve("/bin/sh", ["sh", "-c", "debugfs -c -R 'dump /CONFIGS/mou"...]
10134 10:42:39.116052 +++ exited with 0 +++

Total mount time was 982 sec in this time. But, It took 419 sec for just those two debugfs calls against 982 sec.

These two calls are just reading the CONFIGS/mountdata file from the filesystem, but that could be done directly via libext2fs, along with many of the other operations in libmount_utuls_ldiskfs.c, rather than launching external binaries to do the work. We should take care not to load the whole filesystem metadata, if that can be avoided.

Issue Links

• is related to: LU-1083 tunefs.lustre generates incorrect message when PATH doesn't include /sbin
• is related to: LU-18818 remove mount+open+write+close from ldiskfs_write_ldd()

Comments

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/38027/
Subject: LU-13241 libext2fs: add quota to libext2fs
Project: tools/e2fsprogs
Branch: master-lustre-test
Current Patch Set:
Commit: ab3ebc4782c04ae25568a17e456c29441a0ce044

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38028
Subject: LU-13241 utils: covert to libext2fs for rest of ldiskfs ops
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d1a62adc3449fe9b16c0adc22f77e99393698afe

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38027
Subject: LU-13241 libext2fs: add quota to libext2fs
Project: tools/e2fsprogs
Branch: master-lustre-test
Current Patch Set: 1
Commit: 38122fbe0d40670efbf787141a549875b4186d4f

[Peter Jones]

Landed for 2.14

[Shuichi Ihara]

Although patch was alraedy landed, here is test resutls without/with patch https://review.whamcloud.com/37656/

1 x 1.2PB OST and 'fake' filled up 50% fileystem by patch https://review.whamcloud.com/#/c/37329/.

Average mount time (Tried 3 times)
Without patch: 72.3 sec (77s,70s,70s)
With    patch: 48.6 sec (51s,48s,47s)

So, patch contributed 33% speedup of mount time.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37656/
Subject: LU-13241 utils: use libext2fs for ldiskfs operations
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7dc8aa7e7848f800c54eb18ecd59d665484ce822

[Dongyang Li]

correct, I noticed there's a quota library but it's internal to e2fsprogs. I actually got the ldiskfs_write_ldd working with libext2fs as well, with all the lookup/allocate inode/link parent/write inode dance. However e2fsck complains the quota with the just created fs. maybe we can still write the ldd with libext2fs and use e2fsck to fix the quota at the end of mkfs.lustre?

[Andreas Dilger]

Running e2fsck definitely handles file quota, but maybe only at the end. There is a quota library internal to e2fsprogs, but it isn't included as part of the -devel install.

[Dongyang Li]

Looks like libext2fs doesn't handle quota, so if we use libext2fs to write the ldd, e2fsck will complain about inconsistent usage and quota accounting on the target, so I left the write part out.

we also need to install e2fsprogs-devel on the build box from now on, I'm not sure if Jenkins is setup that way.

 

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37656
Subject: LU-13241 utils: use libext2fs for ldiskfs operations
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bd12ea6a75e185f7d7c457164a0e0c528c26a824