[LU-10401] sanity test_133g: timeout during MDT mount

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.12.0
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run:

Info required for matching: sanity 133g
Info required for matching: sanity 133h

Issue Links

• duplicates: LU-11761 blocked MDT mount and high cpu usage from lodXXXX_recYYYY threads
• is duplicated by: LU-13613 sanity/133h fails
• is related to: LU-7688 sanity test_133g: test failed to respond and timed out
• is related to: LU-14789 sanity 133f and 133g are no longer effective
• is related to: LU-9864 sanity: test_103b failed at ldlm_resource_getref+0x29/0x110
• is related to: LU-8130 Migrate from libcfs hash to rhashtable
• is related to: LU-13558 upgrade e2fsprogs to 1.45.5

Comments

[Yang Sheng]

From log:

[ 8673.618423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.lustre-OST0006-osc-ffff8b2112ff4000.rpc_stats | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.637811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param timeout | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.656550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_backoff | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.675441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_max_delay_centisecs | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8674.097330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity test_133h: @@@@@@ FAIL: files do not end with newline: - 
[ 8674.515181] Lustre: DEBUG MARKER: sanity test_133h: @@@@@@ FAIL: files do not end with newline: -

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38699
Subject: LU-10401 tests: print correct filename while failed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: beba18261bb3f7b1f0bd1b18163dc80b8b9414a5

[Andreas Dilger]

YS, can you please push a patch to fix FILENAME so we can see which filename is having the problem.

[Yang Sheng]

It is my fault. Since awk get input from stdin, So the FILENAME is '-'. But i want to know why this test case is exist? Looks like it only check whether output a '\n' char for get_param. Does it make sense?

[Andreas Dilger]

After the latest patch landing, I'm seeing a couple of places where sanity test_133h is failing:
https://testing.whamcloud.com/test_sets/0a1bd601-7de7-4031-974e-bc138ca14637
https://testing.whamcloud.com/test_sets/9238d945-2814-403b-8ab4-eb96075b4cd4
https://testing.whamcloud.com/test_sets/1073e4a6-5e88-410a-8c80-f0cb15585635
https://testing.whamcloud.com/test_sets/0f75133d-b81d-483b-bfc8-2b90699c9fb9

files do not end with newline: -

It is always '-' as the reported filename, so I suspect there is something wrong with how the test was modified, either reporting an error incorrectly, or not printing out the filename properly when a real error is hit. Two of the above failures were with review-ldiskfs-arm and two were on e2fsprogs tiny sessions. It isn't yet clear if these are permanent or intermittent failures, though I suspect they are permanent failures for those configurations, as I can't imagine how this test would be racy.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38567/
Subject: LU-10401 tests: add -F so list_param prints entry type
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1c54733894f81e854363fbd2d49c141842f73ae4

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38679
Subject: LU-10401 test: debug patch
Project: tools/e2fsprogs
Branch: master-lustre
Current Patch Set: 1
Commit: a0a337f38866706829cd0a45b7ec63d97fc4406e

[Yang Sheng]

Looks like it still exist. Further investigating.

[Peter Jones]

Hmm so did the fix for LU-11761 (marked as included in 2.12.3) not work then?

[Yang Sheng]

So from stacktrace, This issue should duplicate with LU-11761.

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38567
Subject: LU-10401 test: add parameter to print entry type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1cd538d7db07365a3edd5ef9af5c260d22a673a4