[LU-11170] sanity test 415 fails with 'rename took N > M sec'

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.12.0, Lustre 2.13.0, Lustre 2.12.3, Lustre 2.14.0, Lustre 2.12.6, Lustre 2.15.0
• Labels: DNE zfs
• Severity: 3
• Assignee: Vitaliy Kuznetsov
• Reporter: Maloo

Description

This issue was created by maloo for James Nunez <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc

sanity test 415 fails for DNE with ZFS with the error

total: 500 open/close in 0.87 seconds: 572.35 ops/second
rename 500 files took 283 sec
 sanity test_415: @@@@@@ FAIL: rename took 283 sec 

So far, this test only fails for ZFS.

This test started failing on 2018-07-03 with logs at https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc

Other test failures at
https://testing.whamcloud.com/test_sets/8de6a208-8945-11e8-9028-52540065bddc
https://testing.whamcloud.com/test_sets/9bb5e252-8e9c-11e8-b0aa-52540065bddc
https://testing.whamcloud.com/test_sets/029c73ea-8e9e-11e8-87f3-52540065bddc

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_415 - rename took 283 sec
sanity test_415 - rename took 154 > 125 sec

Issue Links

• is blocking: LU-12336 Update ZFS Version to 0.8.2
• is duplicated by: LU-11225 sanity test_415: rename took 176 sec
• is duplicated by: LU-12697 sanity test_415: rename took 133 sec
• is duplicated by: LU-14103 sanity test_415: sanity test_415: @@@@@@ FAIL: rename took 101 sec
• is duplicated by: LU-16237 sanity test_415: rename took 130 sec
• is duplicated by: LU-12843 sanity test_415: rename took 396 sec
• is related to: LU-11102 lock revoke may not take effect

Comments

[Serguei Smirnov]

+1 on master: https://testing.whamcloud.com/test_sets/d8e1c14f-849d-46a5-b422-c77955590561

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49724/
Subject: LU-11170 tests: add debugging to sanity/415
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6594babc73851fab335c514cd1fee018425e7bb3

[Qian Yingjin]

+1 on master:
https://testing.whamcloud.com/test_sets/c37adf87-d38a-4a38-afa4-effadab56dd5

[Sergey Cheremencev]

+1 master

https://testing.whamcloud.com/test_sets/55564f7c-ab54-4536-a2e5-1ec9777a2363

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49724
Subject: LU-11170 tests: add debugging to sanity/415
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e33e8906febd161a89bb190648e74dcc3b7414cb

[Andreas Dilger]

Still hitting this about 1-3% of runs during normal testing. Failures usually complete within 140s, one as long as 269s.

The fact that ZFS is also very slow means that the failures may actually be caused by a regression in the COS code, because commits on ZFS can take a long time. I'm going to push a patch to improve the debugability of the patch, hopefully we can identify the source of the problem.

[Alexander Boyko]

master

https://testing.whamcloud.com/test_sets/8ff591d9-c8e4-43ab-84d3-357f79417e08

[Sarah Liu]

on 2.15 aarch64 client
https://testing.whamcloud.com/test_sets/771da1f7-4616-4a2a-b3fc-d0028e98a485

[Andreas Dilger]

+1 on master:
https://testing.whamcloud.com/test_sets/2e6c4a29-6de8-4e85-bc7c-a986208af412

[Peter Jones]

Tried to avoid again

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47341/
Subject: LU-11170 test: increase time limit in sanity test_415
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b974d7d8a9759601b6f851895ee32714ccf56f28