Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
2.10 (and 2.11)
-
3
-
9223372036854775807
Description
There is a performance regression on dir removal.
Server and client : RHEL7.3
Lustre version : 2.10.52
Backend filesystem: ldiskfs
mpirun --allow-run-as-root /work/tools/bin/mdtest -n 5000 -v -d /scratch0/mdtest.out -D -i 3 -p 10 -w 0 -u
SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 89757.381 65618.928 74607.900 10774.356 Directory stat : 320946.433 319888.242 320294.264 465.749 Directory removal : 19028.569 17837.487 18351.200 499.838 Tree creation : 434.446 158.826 318.943 116.860 Tree removal : 27.018 25.210 26.281 0.775
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
Results of patch https://review.whamcloud.com/#/c/29821/ which reverts just LU-7053:
MIB RESULTS MDTEST RESULTS 000: SUMMARY: (of 3 iterations) 000: Operation Max Min Mean Std Dev 000: --------- --- --- ---- ------- 000: Directory creation: 21177.729 16443.602 18498.262 1982.554 000: Directory stat : 232409.626 229897.859 230874.272 1098.955 000: Directory removal : 111897.307 40906.454 75055.964 29044.331 000: File creation : 44061.464 38438.052 41863.997 2454.596 000: File stat : 225941.782 193254.640 210777.086 13448.211 000: File read : 146598.308 86669.775 126562.125 28208.247 000: File removal : 155512.154 106886.927 136397.357 21168.452 000: Tree creation : 120.178 56.129 96.113 28.468 000: Tree removal : 8.906 8.122 8.620 0.354 000:
Dir removal for this run has been "75055.964" (mean) approx 45% higher in performance in comparison to the patch https://review.whamcloud.com/29709 under exactly same conditions and number of iterations.
to verify that I added printk() to osd_idc_find_or_init() and got zero calls to osd_remote_fid() and osd_oi_lookup() during rmdir.
there are few calls to osd_remote_fid() (which I hope to fix in a separate patch), but those weren't introduced with LU-7053
usually it's initialized from other preceding methods (like osd_declare_ref_
{add|del} in https://review.whamcloud.com/29709
with no extra lookups.
Yes, but I'm not talking about the cost of accessing the IDC cache. I mean the extra cost of OI lookup to initialize the IDC entry.
Before LU-7053, when calling osd_index_declare_ea_insert() from orph_declare_index_insert(), we only looked up the FID of mdd->mdd_orphans in the FLDB to determine if it was local or remote (so that we could declare additional credits in the remote case). Now we do osd_idc_find_or_init() which included OI lookup in the local case. I understand that we may need the extra information if osd_index_ea_insert() gets called to add the directory to the orphan dir. But I assume that we are not hitting this case from mdtest. So maybe the difference between FLDB lookup and full IDC initialization (and/or contention effects) could explain the difference.
please, benchmark master with https://review.whamcloud.com/#/c/29821/
this patch reverts just LU-7053.
Andreas, at the moment I don't quite understand how the cache can decrease performance as it's TLS, lockless, tiny and it replaces a lookup in LU cache which in contrast much larger and needs locking. I keep investigating..
Build Version Dir create Dir stat Dir rm b_ieel3_0-89 2.7.16.1 19758 169050 77119 b_ieel3_0-105 2.7.18 21239 167421 73112 b_ieel3_0-214 2.7.20.2 28136 331563 60515 master-3601 2.9.59 21063 223101 11879 master-2607 2.8.59-35 19328 211813 11797 master-3637 2.10.52_83 25987 234954 15033 master 2.10.54 18676 210767 10282 review-51618 2.10.54-20-g66bb2d1 18542 233266 51633
So it looks like the rmdir performance is significantly improved, but the mkdir performance is down a bit from 2.10.53, but about on par with 2.10.54. It looks like the file create/stat/unlink performance is a bit lower vs. 2.10.54, but definitely still a lot better than 2.7.64.
Alex, would it make sense to only add directory objects to the cache, instead of adding all objects? That may give us the best of both worlds.
Performance results with patch above:
MDTEST RESULTS 000: SUMMARY: (of 3 iterations) 000: Operation Max Min Mean Std Dev 000: --------- --- --- ---- ------- 000: Directory creation: 24200.233 15693.923 18542.903 4000.371 000: Directory stat : 235384.333 230225.191 233266.050 2204.926 000: Directory removal : 82809.228 30178.490 51633.084 22559.256 000: File creation : 50508.075 33979.054 41331.172 6870.202 000: File stat : 232294.752 226222.165 228931.104 2521.978 000: File read : 124620.945 110484.961 119398.361 6333.672 000: File removal : 128159.287 77027.802 106534.012 21605.386 000: Tree creation : 124.922 65.746 103.702 26.901 000: Tree removal : 9.709 7.841 8.650 0.783
Results for dir removal with this patch have surely improved but still not to the mark where they were before LU-7053 was landed.
Test run for Lustre version just before landing of LU-7053 showed performance number for Dir removal as "92029.569" (mean) .
thanks for the data. I've updated https://review.whamcloud.com/29709
please benchmark master with that. the patch collects additional stats and dump it at umount in the following form:
+ if (atomic_read(&o->od_idc_remotes) ||
+ atomic_read(&o->od_idc_oi))
+ printk("%s: %d checks for remote, %d OI lookups\n",
+ o->od_svname,
+ atomic_read(&o->od_idc_remotes),
+ atomic_read(&o->od_idc_oi));
please attach it here if printed.
thanks in advance.
I'm working on a followup patch to optimize calls to osd_remote_fid(), this isn't directly related to
LU-7053, but hopefully can fix the issue.