[LU-9972] Performance regressions on unique directory removal

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Environment: 2.10 (and 2.11)
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Shuichi Ihara (Inactive)

Description

There is a performance regression on dir removal.

Server and client : RHEL7.3
Lustre version : 2.10.52
Backend filesystem: ldiskfs

mpirun --allow-run-as-root /work/tools/bin/mdtest -n 5000 -v -d /scratch0/mdtest.out -D -i 3 -p 10 -w 0 -u

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      89757.381      65618.928      74607.900      10774.356
   Directory stat    :     320946.433     319888.242     320294.264        465.749
   Directory removal :      19028.569      17837.487      18351.200        499.838
   Tree creation     :        434.446        158.826        318.943        116.860
   Tree removal      :         27.018         25.210         26.281          0.775

Issue Links

• is related to: LU-7053 Do not use osd_object_find in osd_index_ea_insert

Comments

[Andreas Dilger]

Build          Version             Dir create   Dir stat   Dir rm
b_ieel3_0-89   2.7.16.1                 19758     169050    77119 
b_ieel3_0-105  2.7.18                   21239     167421    73112
b_ieel3_0-214  2.7.20.2                 28136     331563    60515
master-3601    2.9.59                   21063     223101    11879
master-2607    2.8.59-35                19328     211813    11797
master-3637    2.10.52_83               25987     234954    15033
master         2.10.54                  18676	  210767    10282
review-51618   2.10.54-20-g66bb2d1      18542     233266    51633

So it looks like the rmdir performance is significantly improved, but the mkdir performance is down a bit from 2.10.53, but about on par with 2.10.54. It looks like the file create/stat/unlink performance is a bit lower vs. 2.10.54, but definitely still a lot better than 2.7.64.

Alex, would it make sense to only add directory objects to the cache, instead of adding all objects?  That may give us the best of both worlds.

[Saurabh Tandan (Inactive)]

Performance results with patch above:

MDTEST RESULTS
000: SUMMARY: (of 3 iterations)
000:    Operation                      Max            Min           Mean        Std Dev
000:    ---------                      ---            ---           ----        -------
000:    Directory creation:      24200.233      15693.923      18542.903       4000.371
000:    Directory stat    :     235384.333     230225.191     233266.050       2204.926
000:    Directory removal :      82809.228      30178.490      51633.084      22559.256
000:    File creation     :      50508.075      33979.054      41331.172       6870.202
000:    File stat         :     232294.752     226222.165     228931.104       2521.978
000:    File read         :     124620.945     110484.961     119398.361       6333.672
000:    File removal      :     128159.287      77027.802     106534.012      21605.386
000:    Tree creation     :        124.922         65.746        103.702         26.901
000:    Tree removal      :          9.709          7.841          8.650          0.783

Results for dir removal with this patch have surely improved but still not to the mark where they were before LU-7053 was landed.
Test run for Lustre version just before landing of LU-7053 showed performance number for Dir removal as "92029.569" (mean) .

[Alex Zhuravlev]

please, try with the patch above.

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29709
Subject: LU-9972 osd: cache OI mapping in dt_ref_

{del|add}

()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ccb8803749e9234e4bd6e5e451a3333d6b17e594

[Alex Zhuravlev]

I tried to revert LU-7053 on master and got no difference for rmdir:
with LU-7053: total: 100000 unlinks in 16 seconds: 6250.000000 unlinks/second
without LU-7053: total: 100000 unlinks in 16 seconds: 6250.000000 unlinks/second
the load was simulated with createmany/unlinkmany

going to proceed with mdtest, but that will take some time.

[Andreas Dilger]

I've updated the results to show the commit-order test results for the bisect (not the bisect order), to show there is a clear break between LU-7408 and the next patch LU-7053.

[John Hammond]

There must have been more runs than just these if you were able to isolate https://review.whamcloud.com/#/c/17092/.

[Saurabh Tandan (Inactive)]

There was approximately a drop of 90% in performance fir "Dir removal" for "mdtestfpp" results from tag 2.7.65. Following is the data for all the runs:

Tag                             Dir removal
2.7.56                          18298
2.7.57                         121954
2.7.61                          64849
2.7.64                         111655 good
2.7.64-g63a3e412 (LU-7419)      74384 good
2.7.64-gc965fc8a (LU-7450)      72374 good
2.7.64-g6765d785 (LU-7408)      92029 good
2.7.64-g9ae3a289 (LU-7053)      11517 bad
2.7.64-g0d3a07a8 (LU-7430)      15114 bad
2.7.64-g959f8f78 (LU-7573)      11530 bad 
2.7.65                          11375 bad
2.7.66                          11403 bad
2.10.53                         12473
2.10.54                          9649

[Peter Jones]

Alex

I daresay that Saurabh may elaborate but I understand that he has found that your patch LU-7053 (osd: don't lookup object at insert https://review.whamcloud.com/#/c/17092/) is the one that introduced the performance regression with directory removal

Do you have any ideas on how to avoid this?

Peter

[Saurabh Tandan (Inactive)]

Still working on it Shuichi, will soon post some findings.

[Shuichi Ihara (Inactive)]

Any progress on finding regression point?