[LU-4367] unlink performance regression on lustre-2.5.52 client

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.5.0
• Labels: HB

Epic/Theme

Performance

• Severity: 2
• Assignee: Lai Siyao
• Reporter: Shuichi Ihara (Inactive)

Description

lustre-2.5.52 client (and maybe more old client as well) causes metadata performance (unlink files in the single shared directory) regression.
Here is test results on lustre-2.5.52 clients and lustre-2.4.1 clients. lustre-2.5.52 is running on all servers.

1 x MDS, 4 x OSS (32 x OST) and 16 clients(64 processs, 20000 files per process)

lustre-2.4.1 client

4.1-take2.log
-- started at 12/09/2013 07:31:29 --

mdtest-1.9.1 was launched with 64 total task(s) on 16 node(s)
Command line used: /work/tools/bin/mdtest -d /lustre/dir.0 -n 20000 -F -i 3
Path: /lustre
FS: 1141.8 TiB   Used FS: 0.0%   Inodes: 50.0 Mi   Used Inodes: 0.0%

64 tasks, 1280000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      58200.265      56783.559      57589.448        594.589
   File stat         :     123351.857     109571.584     114223.612       6455.043
   File read         :     109917.788      83891.903      99965.718      11472.968
   File removal      :      60825.889      59066.121      59782.774        754.599
   Tree creation     :       4048.556       1971.934       3082.293        853.878
   Tree removal      :         21.269         15.069         18.204          2.532

-- finished at 12/09/2013 07:34:53 --

lustre-2.5.5.2 client

-- started at 12/09/2013 07:13:42 --

mdtest-1.9.1 was launched with 64 total task(s) on 16 node(s)
Command line used: /work/tools/bin/mdtest -d /lustre/dir.0 -n 20000 -F -i 3
Path: /lustre
FS: 1141.8 TiB   Used FS: 0.0%   Inodes: 50.0 Mi   Used Inodes: 0.0%

64 tasks, 1280000 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      58286.631      56689.423      57298.286        705.112
   File stat         :     127671.818     116429.261     121610.854       4631.684
   File read         :     173527.817     158205.242     166676.568       6359.445
   File removal      :      46818.194      45638.851      46118.111        506.151
   Tree creation     :       3844.458       2576.354       3393.050        578.560
   Tree removal      :         21.383         18.329         19.844          1.247

-- finished at 12/09/2013 07:17:07 --

46K ops/sec (lusre-2.5.52) vs 60K ops/sec (lustre-2.4.1). 25% performance drops on Lustre-2.5.52 compared to Lustre-2.4.1.

Issue Links

• is related to: LU-4906 rm -rf triggers too much MDS_READPAGE
• is related to: LU-5197 A performance regression of "FileRead" metadata operation
• is related to: LU-8019 Openlock breakage
• is related to: LU-5426 Add more controls for open file cache

Comments

[Shuichi Ihara (Inactive)]

I'm testing patches. will post results shortly.

[Andreas Dilger]

Ihara, did you get a chance to test if 10398 fixes the unlink regression? We are ready to land that patch.

[Andreas Dilger]

Lai should confirm, but I think the most important patch for addressing the unlink regression is http://review.whamcloud.com/10398 so that one should be tested first.

There is also a potential improvement in http://review.whamcloud.com/9696 that is next, but it doesn't affect unlink. I think the http://review.whamcloud.com/9697 is too complex to land for 2.6.0 at this point, but if it gives a significant improvement then it could be landed for 2.7.0 and IEEL.

[Shuichi Ihara (Inactive)]

Hi Lai,
it seems that there are several updates after you posted initial patches. please adivse me which patches should be applied?

[Lai Siyao]

Patch to combine close in unlink RPC: http://review.whamcloud.com/#/c/10398/

Ihara, could you apply this only and get results from mdtest?

[Lai Siyao]

I've thought of that, but considering the complication of open replay, and possibly SOM, I think it's not a trivial work. I'll think about it more and do some test later (maybe next week).

[Andreas Dilger]

It might be possible to combine the close and unlink RPCs (unlink with close flag, or close with unlink flag?) so that the number of RPCs is actually reduced? We already do something similar with early lock cancellation, so it might be possible to do something similar with the close.

[Lai Siyao]

The root cause is that revalidate(IT_OPEN) enqueued open lock, so that close is deferred to unlink which caused unlink performance drop, but totally there is no extra RPC. I don't find a clear way to handle this, so I think if we can improve open and stat performance a lot, it's worthwhile keeping the status quo.

[Andreas Dilger]

Lai, it looks like the patches http://review.whamcloud.com/9696 and http://review.whamcloud.com/9697 are improving the open performance, but do not address the unlink performance. Is there something that can be done to improve the unlink performance back to the 2.5.0 level so that 2.6.0 does not have a performance regression?

[Lai Siyao]

Thanks Ihara, patches updated, previously I only tested mdtest, and didn't do a full test because they are intended to get mdtest performance data, and may not be final patches yet, sorry for the trouble made.

[Shuichi Ihara (Inactive)]

this is debugfile when the problem happens.

echo "+trace" > /proc/sys/lnet/debug
lctl debug_daemon start /tmp/debuglog 100
touch /tmp/a
cp /tmp/a /lustre
lctl debug_daemon stop
echo "-trace" > /proc/sys/lnet/debug