[LU-5621] Performance regression in 2.6 branch on file operations to shared directory

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0
• Type: Bug
• Priority: Major
• Reporter: Shuichi Ihara (Inactive)
• Assignee: Peter Jones
• Resolution: Won't Fix

Environment

lustre-2.6.52

• Severity: 2
• Epic: metadata, performance

Description

We found a performance regression in master branch on our metadata scalability and regression testing.
Lustre-2.6 server is more than 50% slower against lustre-2.5 server on file creation/removal to shared directory. Here is test results.

mdtest -n $((1280000/NP)) -i 3 -p 10 -d /lustre_{0-$NP}/mdtest.out -F -C -r

File creation(Shared directory)	2.5 server	2.6 server
32 threads	69035	33354
64 threads	80798	30619
128 threads	69896	29956
256 threads	64860	29951
512 threads	70733	30070
1024 threads	73064	30540

File removal(Shared directory)	2.5 server	2.6 server
32 threads	61087	52609
64 threads	82981	46370
128 threads	72802	37697
256 threads	78388	35324
512 threads	81033	41058
1024 threads	79155	40544

Issue Links

• is related to: LU-5663 mds-survey performance regress on master

Comments

[Di Wang]

What is the client version? same for two tests?

[Shuichi Ihara (Inactive)]

tested with 2.5 clients for both testing. I also have tested 2.6 client for 2.6 server testing(not 2.5 server test), but it was same results.

[Jinshan Xiong (Inactive)]

John Hammond has ever submitted a patch about this but I don't know the current status. Please refresh my memory, John.

[Di Wang]

Not sure it is a known issue or not. If it is not, it is worth to run mds-survey to see whether the problem is in LOD/OSD.

[Shuichi Ihara (Inactive)]

OK, but "git bisect" would be easy way to find what exactly commit affects this regression. I will do it.

[Peter Jones]

Once we see the results of the git bisect I'll work out who to assign this to

[John Hammond]

Jinshan, do you mean the following?

commit 708d85a652a77f85153790e6cca1b7a2b91947cf
Author: John L. Hammond <[e-mail]>
Date:   Thu Jan 30 11:07:13 2014 -0600

    LU-4398 mdt: acquire an open lock for write or execute
    
    In mdt_object_open_lock() opens for write or execute will always
    acquire an open lock of the appropriate mode so that any conflicting
    cached open locks on other clients will be canceled. Add a regression
    test to sanityn.sh.
    
    Signed-off-by: John L. Hammond <[e-mail]>
    Change-Id: I8092bca4c418ec99a25584abdfb635ffec19a26e
    Reviewed-on: http://review.whamcloud.com/9063
    Reviewed-by: Jinshan Xiong <[e-mail]>
    Tested-by: Jenkins
    Tested-by: Maloo <[e-mail]>
    Reviewed-by: Mike Pershin <[e-mail]>
    Reviewed-by: Oleg Drokin <[e-mail]>

That was landed after 2.5.0 and reverted before 2.6.0.

[Jinshan Xiong (Inactive)]

This issue may be related to LU-5663.

[Di Wang]

Ihara, could you please try the patch in LU-5663 to see whether it can improve the performance of your test. That patch eliminates the overhead of striped directory (brought in 2.6) in the create object patch, then I can see whether striped directory cause this performance regression here.

[Di Wang]

Iharah, althogh this patch does not show any difference in Jinshan's test, it does not mean it would not help yours. Because his test is more about single (or few) thread on each directory. This temporary patch will probably be more helpful for shared directory. Thank you!

[Di Wang]

http://review.whamcloud.com/#/c/12195 Not sure if this will help much, but worth to try.

[Alex Zhuravlev]

is it possible to get profiling for the both versions?

[Peter Jones]

I think that this issue is now out of date