[LU-11152] sanity test_133g: ost1 find /proc/fs/lustre/ /proc/sys/lnet/ /proc/sys/lustre/ failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.12.0
• Fix Version/s: Lustre 2.14.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Andreas Dilger
• Resolution: Fixed
• Severity: 3

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/28765bb2-8831-11e8-9e83-52540065bddc

test_133g failed with the following error:

$'ost1 find /proc/fs/lustre/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed'

This seems to be only failing on review-dne-zfs-part-1 and review-dne-part-1.

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_133g - $'ost1 find /proc/fs/lustre/n/sys/fs/lustre/n/sys/kernel/debug/lnet/n/sys/kernel/debug/lustre/ failed'

Issue Links

• is related to: LU-9700 Interop 2.9.0<->master sanity test_13...
• is related to: LU-10038 sanity test 133g fails with “ '$'mds1...
• is related to: LU-11552 improper FMR/FastReg pool cleanup
• is related to: LU-11735 o2iblnd - bad check for fmr

Comments

[Andreas Dilger]

This is similar to LU-9700, but no interop involved. LU-10038 was reported fixed for 2.11, but this problem is still being seen (about once a day in the past two weeks).

[Andreas Dilger]

Also seeing some cases like:

trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0000/exports/10.9.1.249@tcp’: No such file or directory
trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0005/exports/10.9.1.249@tcp’: No such file or directory
trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0006/exports/10.9.1.249@tcp’: No such file or directory
trevis-57vm8: find: ‘/proc/fs/lustre/obdfilter/lustre-OST0007/exports/10.9.1.249@tcp’: No such file or directory

[John Hammond]

This is a normal race which we handle by passing the -ignore_readdir_race flag to find. See LU-10038. However, in find 4.5.11 which is included in CentOS 7.3-7.5 the -ignore_readdir_race flag is ineffective. See https://bugs.centos.org/view.php?id=13685.

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32934
Subject: LU-11152 test: work around find bug in sanity 133[fg]
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: aa345cb2e7eeba0699157a1a16e242e9a1a54a4e

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/32934/
Subject: LU-11152 test: work around find bug in sanity 133[fg]
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: edd1b078941b9cc97e39e0b54b9daabe0e6f2792

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch:
https://review.whamcloud.com/33408
Subject: LU-11152 lnd: correctly cleanup the FMR/FastReg pools
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5ea8405a7be061bd485c2f94c09d14ccfbac48b0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch
https://review.whamcloud.com/33408/
Subject: LU-11152 lnd: test fpo_fmr_poool pointer instead of special bool
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9b790ba0f5606c0a91563828fa43f5e4ae210425

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch:
https://review.whamcloud.com/33802
Subject: Revert "LU-11152 lnd: test fpo_fmr_poool pointer instead of special bool"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b169980b805b78df9b05e10079d27c4a6d3dbbc1

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch
https://review.whamcloud.com/33802/
Subject: Revert "LU-11152 lnd: test fpo_fmr_poool pointer instead of special bool"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a65d072fa45fc90c2fc74b61d214de79c0bf33e5

[Peter Jones]

Reverting because this has introduced LU-11735

[Cory Spitz]

It seems that we have a bit of mix-up here.

9b790ba0f5606c0a91563828fa43f5e4ae210425 (LU-11152 lnd: test fpo_fmr_poool pointer instead of special bool) was landed for "LU-11152", but it was really intended for LU-11552. The FMR pool stuff shouldn't have anything to do with this ticket.

Then it was reverted with:
commit a65d072fa45fc90c2fc74b61d214de79c0bf33e5
Author: Amir Shehata <[e-mail]>
Date: Thu Dec 6 20:52:22 2018 +0000

Revert "LU-11152 lnd: test fpo_fmr_poool pointer instead of special bool"

This reverts commit 9b790ba0f5606c0a91563828fa43f5e4ae210425.

And then this ticket was implicated.

I don't know how to clean up this mess, but I'm going to post a duplicate comment in LU-11552. Maybe we don't need to do anything special other than to add a helpful comment to the commit message if/when LU-11735 or LU-11552 are addressed.

[Andreas Dilger]

Cory, thanks for pointing out the discrepancy here. Too bad this wasn't caught before these patches were landed, but it isn't possible to change the commit comments in Git.