[LU-9972] Performance regressions on unique directory removal

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Environment: 2.10 (and 2.11)
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Shuichi Ihara (Inactive)

Description

There is a performance regression on dir removal.

Server and client : RHEL7.3
Lustre version : 2.10.52
Backend filesystem: ldiskfs

mpirun --allow-run-as-root /work/tools/bin/mdtest -n 5000 -v -d /scratch0/mdtest.out -D -i 3 -p 10 -w 0 -u

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      89757.381      65618.928      74607.900      10774.356
   Directory stat    :     320946.433     319888.242     320294.264        465.749
   Directory removal :      19028.569      17837.487      18351.200        499.838
   Tree creation     :        434.446        158.826        318.943        116.860
   Tree removal      :         27.018         25.210         26.281          0.775

Issue Links

• is related to: LU-7053 Do not use osd_object_find in osd_index_ea_insert

Comments

[Saurabh Tandan (Inactive)]

Results of patch https://review.whamcloud.com/#/c/29821/ which reverts just LU-7053:

MIB RESULTS
MDTEST RESULTS
000: SUMMARY: (of 3 iterations)
000:    Operation                      Max            Min           Mean        Std Dev
000:    ---------                      ---            ---           ----        -------
000:    Directory creation:      21177.729      16443.602      18498.262       1982.554
000:    Directory stat    :     232409.626     229897.859     230874.272       1098.955
000:    Directory removal :     111897.307      40906.454      75055.964      29044.331
000:    File creation     :      44061.464      38438.052      41863.997       2454.596
000:    File stat         :     225941.782     193254.640     210777.086      13448.211
000:    File read         :     146598.308      86669.775     126562.125      28208.247
000:    File removal      :     155512.154     106886.927     136397.357      21168.452
000:    Tree creation     :        120.178         56.129         96.113         28.468
000:    Tree removal      :          8.906          8.122          8.620          0.354
000:

Dir removal for this run has been "75055.964" (mean) approx 45% higher in performance in comparison to the patch https://review.whamcloud.com/29709 under exactly same conditions and number of iterations.

[Alex Zhuravlev]

to verify that I added printk() to osd_idc_find_or_init() and got zero calls to osd_remote_fid() and osd_oi_lookup() during rmdir.
there are few calls to osd_remote_fid() (which I hope to fix in a separate patch), but those weren't introduced with LU-7053

[Alex Zhuravlev]

usually it's initialized from other preceding methods (like osd_declare_ref_

{add|del}

in https://review.whamcloud.com/29709
with no extra lookups.

[John Hammond]

Yes, but I'm not talking about the cost of accessing the IDC cache. I mean the extra cost of OI lookup to initialize the IDC entry.

[Alex Zhuravlev]

IDC is lockless and per-thread while FLDB is a shared structure.

[John Hammond]

Before LU-7053, when calling osd_index_declare_ea_insert() from orph_declare_index_insert(), we only looked up the FID of mdd->mdd_orphans in the FLDB to determine if it was local or remote (so that we could declare additional credits in the remote case). Now we do osd_idc_find_or_init() which included OI lookup in the local case. I understand that we may need the extra information if osd_index_ea_insert() gets called to add the directory to the orphan dir. But I assume that we are not hitting this case from mdtest. So maybe the difference between FLDB lookup and full IDC initialization (and/or contention effects) could explain the difference.

[Alex Zhuravlev]

please, benchmark master with https://review.whamcloud.com/#/c/29821/
this patch reverts just LU-7053.

[Alex Zhuravlev]

Andreas, at the moment I don't quite understand how the cache can decrease performance as it's TLS, lockless, tiny and it replaces a lookup in LU cache which in contrast much larger and needs locking. I keep investigating..

[Andreas Dilger]

Build          Version             Dir create   Dir stat   Dir rm
b_ieel3_0-89   2.7.16.1                 19758     169050    77119 
b_ieel3_0-105  2.7.18                   21239     167421    73112
b_ieel3_0-214  2.7.20.2                 28136     331563    60515
master-3601    2.9.59                   21063     223101    11879
master-2607    2.8.59-35                19328     211813    11797
master-3637    2.10.52_83               25987     234954    15033
master         2.10.54                  18676	  210767    10282
review-51618   2.10.54-20-g66bb2d1      18542     233266    51633

So it looks like the rmdir performance is significantly improved, but the mkdir performance is down a bit from 2.10.53, but about on par with 2.10.54. It looks like the file create/stat/unlink performance is a bit lower vs. 2.10.54, but definitely still a lot better than 2.7.64.

Alex, would it make sense to only add directory objects to the cache, instead of adding all objects?  That may give us the best of both worlds.

[Saurabh Tandan (Inactive)]

Performance results with patch above:

MDTEST RESULTS
000: SUMMARY: (of 3 iterations)
000:    Operation                      Max            Min           Mean        Std Dev
000:    ---------                      ---            ---           ----        -------
000:    Directory creation:      24200.233      15693.923      18542.903       4000.371
000:    Directory stat    :     235384.333     230225.191     233266.050       2204.926
000:    Directory removal :      82809.228      30178.490      51633.084      22559.256
000:    File creation     :      50508.075      33979.054      41331.172       6870.202
000:    File stat         :     232294.752     226222.165     228931.104       2521.978
000:    File read         :     124620.945     110484.961     119398.361       6333.672
000:    File removal      :     128159.287      77027.802     106534.012      21605.386
000:    Tree creation     :        124.922         65.746        103.702         26.901
000:    Tree removal      :          9.709          7.841          8.650          0.783

Results for dir removal with this patch have surely improved but still not to the mark where they were before LU-7053 was landed.
Test run for Lustre version just before landing of LU-7053 showed performance number for Dir removal as "92029.569" (mean) .

[Alex Zhuravlev]

please, try with the patch above.