[LU-10401] sanity test_133g: timeout during MDT mount

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.12.0
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run:

Info required for matching: sanity 133g
Info required for matching: sanity 133h

Issue Links

• duplicates: LU-11761 blocked MDT mount and high cpu usage from lodXXXX_recYYYY threads
• is duplicated by: LU-13613 sanity/133h fails
• is related to: LU-7688 sanity test_133g: test failed to respond and timed out
• is related to: LU-14789 sanity 133f and 133g are no longer effective
• is related to: LU-9864 sanity: test_103b failed at ldlm_resource_getref+0x29/0x110
• is related to: LU-8130 Migrate from libcfs hash to rhashtable
• is related to: LU-13558 upgrade e2fsprogs to 1.45.5

Comments

[Andreas Dilger]

Patch has landed to master for 2.14. Patches will need to be rebased if they are failing because of this issue.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/38699/
Subject: LU-10401 procs: print new line based on distro
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0130d7019534611ff05a29fad657226d2eb6b79e

[Bruno Faccini (Inactive)]

Oops my bad, Peter and Yang Sheng, I had missed the fix part of #38699. Thanks for your help.

[Peter Jones]

I am just going on that this has been used successfully for a couple of other patches that hit these failures. This patch is in master-next ATM so should land shortly.

[Yang Sheng]

Hi， Bruno，

The latest 38699 is a fixing patch rather than debug patch. So please rebase your patch on top of it to overcome sanity-133h failure.

Thanks,
YangSheng

[Bruno Faccini (Inactive)]

Peter, thanks for your update. But looking at #38699, it looks as a pure debug patch. And according to the current failure rate of sanity/test_133h it should be easy for Yang Sheng to get the associated debug infos now.

[Peter Jones]

I literally just hit post on a comment in Gerrit about this. There is a patch - https://review.whamcloud.com/#/c/38699/ - that you could try rebasing on top of

[Bruno Faccini (Inactive)]

After rebasing with current master, seems that patch-set #32 of my change @ https://review.whamcloud.com/#/c/35856/32 triggers LU-10401 almost solid !!!! And according to this weeks stats for sanity/test_133h a lot of others auto-testing sessions are also affected.
Could the priority for this issue be raised or at least sanity/test_133h been skipped ?

[Arshad Hussain]

+1 On Master : https://testing.whamcloud.com/test_sets/619d15cb-a12b-489b-a085-cb473ad81772

[Yang Sheng]

From log:

[ 8673.618423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.lustre-OST0006-osc-ffff8b2112ff4000.rpc_stats | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.637811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param timeout | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.656550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_backoff | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8673.675441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_max_delay_centisecs | awk -v FS=\v -v RS=\v\v 'END { if(NR>0 && $NF !~ /.*\n$/) print FILENAME}'
[ 8674.097330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity test_133h: @@@@@@ FAIL: files do not end with newline: - 
[ 8674.515181] Lustre: DEBUG MARKER: sanity test_133h: @@@@@@ FAIL: files do not end with newline: -