Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
9223372036854775807
Description
during Lustre server startup few small files need to be updated (e.g. config backup).
at this point buddy/bitmap cache is empty but mballoc wants to find a big chunk of free space for group preallocation and reads bitmaps one by one.
sometimes this can take a very long time.
one possisble workaround is to disable preallocation during mount.
a long term plan is to limit scanning and prefetch bitmaps, but this needs more efforts and will be tracked separately (LU-12970)
Attachments
Issue Links
Activity
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37155/
Subject: LU-12988 osd: do not use preallocation during mount
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 2331cd1fa178b348d8aa048abbb5160ac9353461
Do you also have Artem's mballoc patch applied?
yes, AFAICS. and that didn't help, probably fillness wasn't high enough. I'll check tomorrow.
Do you also have Artem's mballoc patch applied? If not, should that be reverted from master, or do you think it makes sense to keep, but with lower thresholds (eg. 15%, 10%, 5% or whatever)?
both https://review.whamcloud.com/#/c/36891/ and https://review.whamcloud.com/#/c/36893/ were used and need refresh, working on this..
Alex, which patch(es) we're used for you test? Can you please update the commit message for you patch with these test results and we can get them landed.
on 85% filled fs using "slow" devmapper with 5ms IO delay:
/dev/mapper/dm-slow 124T 100T 18T 85% /mnt/ost2
tried to mount with non-patched ldiskfs, but interrupted after about half an hour:
# time mount -t lustre /dev/mapper/dm-slow /mnt/ost2 ^C real 27m42.190s user 0m0.000s sys 0m0.002s procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 0 13311820 398560 6025368 0 0 200 0 67 332 0 0 50 50 0 0 1 0 13311200 398760 6025680 0 0 200 0 65 327 0 0 50 50 1 0 1 0 13310456 398960 6026220 0 0 200 0 66 338 0 0 50 50 0 0 1 0 13309836 399160 6026620 0 0 200 0 162 426 0 0 50 50 0 0 1 0 13309092 399360 6027036 0 0 200 0 165 422 0 0 50 50 1 0 1 0 13308472 399560 6027428 0 0 200 0 169 434 0 0 50 50 0 0 1 0 13308100 399760 6027912 0 0 200 0 164 421 0 0 50 50 0 0 1 0 13307356 399960 6028240 0 0 200 0 168 439 0 0 50 50 0
according to vmstat mount was going to take ~11 hours.
now with patched ldiskfs with cache wiped by echo 3 >/proc/sys/vm/drop_caches:
# time mount -t lustre /dev/mapper/dm-slow /mnt/ost2 real 0m20.364s user 0m0.192s sys 0m1.103s
echo "0 134217728000 delay /dev/loop0 0 5" | dmsetup create dm-slow
so now:
# dd if=/dev/loop0 of=/dev/null bs=4k count=12800 12800+0 records in 12800+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.0662961 s, 791 MB/s # dd if=/dev/mapper/dm-slow of=/dev/null bs=4k count=12800 12800+0 records in 12800+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.05419 s, 12.9 MB/s
trying to start Lustre on this virtual device..
bzzz, do you have results of comparing mount time with https://review.whamcloud.com/#/c/36704/ and without it? Please, share if you have.
Artem's patch wasn't used. I'm thinking of few policies.. will try to make an overview soon.
Are these results with or without Artem's patch https://review.whamcloud.com/35180 "LU-12103 ldiskfs: don't search large block range if disk full"? That would skip cr=0 and cr=1 immediately because of filesystem fullness. That may not be ideal for a real-world filesystem because the 80-20 block full-free pattern is unlikely to be seen in real life, but at the same time it would also find the groups that have free chunks and allocate more there.
The one improvement that would be possible with Artem's patch would be to not make it strictly based on filesystem fullness, but also using cX_failed to decide when the filesystem is too fragmented.
bzzz I hope I am wrong...
I also get such optimistic results with my "adjusting table" script, but then I had realized that only FIRST MOUNT is slow. I recreated the partition and when mount became slow again even with my optimizations. bzzz, could you check your performance improvement on the fresh-created filesystem?