[LU-13112] sanity test 228b fails with “ Fail to df. “

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.14.0
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Cannot Reproduce
• Labels: arm

Environment

ARM clients

• Severity: 3

Description

sanity test_228b fails with “ Fail to df. “ sanity test 228b started failing with this error on 2019-12-25 and only fails for ARM.

Looking at the failure at https://testing.whamcloud.com/test_sets/aa1beab0-290c-11ea-adca-52540065bddc, we see ‘df’ fail with

Started lustre-MDT0000
df: '/mnt/lustre': No such device
 sanity test_228b: @@@@@@ FAIL: Fail to df.

Looking at the client (vm9) console log, we see an error

[ 9064.064496] LustreError: 18300:0:(lmv_obd.c:1261:lmv_statfs()) lustre-MDT0000-mdc-ffff800065d47800: can't stat MDS #0: rc = -19
[ 9064.715178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity test_228b: @@@@@@ FAIL: Fail to df. 

There are several examples of this failure
https://testing.whamcloud.com/test_sets/aa302450-290a-11ea-b0f4-52540065bddc
https://testing.whamcloud.com/test_sets/19dedb6e-28ea-11ea-b0f4-52540065bddc
https://testing.whamcloud.com/test_sets/10b8929c-288a-11ea-bb75-52540065bddc

Comments

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37138
Subject: LU-13112 tests: determine cause of failure
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 82398d9abc1ceb493acac230e27b976355901e51

[Andreas Dilger]

+1 on master x86_64: https://testing.whamcloud.com/test_sets/cf84b726-3b33-11ea-b0f4-52540065bddc

[Andreas Dilger]

I checked current test results and 54/54 passed on aarch64 in the past week. 

[James Nunez (Inactive)]

I've seen this failure once for non-arm testing. The failure is for interop testing, 2.13.0 servers with master (2.14.0) clients, at https://testing.whamcloud.com/test_sets/cac2fa87-490a-4e0e-a724-ac78b7976118 .

As Andreas noted above, we aren't seeing this test fail for ARM client testing anymore. The last time we saw this test fail for ARM was on 10 FEB 2020.

[Xinliang Liu]

Run 100 times in local Arm all-in-one CentOS 8 environment.  All pass, this issue maybe caused/impacted by other test cases.

[root@test-01 tests]# PTLDEBUG=-1  RUNAS_ID="1000" RUNAS_GID=1000 ./auster -i 100  -vsr sanity --only 228b
....
/tmp/test_logs/2021-10-26/034930/74/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (25s)
/tmp/test_logs/2021-10-26/034930/75/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/76/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/77/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/78/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/79/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/80/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/81/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/82/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/83/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/84/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/85/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/86/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/87/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/88/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/89/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (25s)
/tmp/test_logs/2021-10-26/034930/90/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/91/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/92/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/93/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (34s)
/tmp/test_logs/2021-10-26/034930/94/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (23s)
/tmp/test_logs/2021-10-26/034930/95/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/96/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (27s)
/tmp/test_logs/2021-10-26/034930/97/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
/tmp/test_logs/2021-10-26/034930/98/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (25s)
/tmp/test_logs/2021-10-26/034930/99/sanity.suite_log.beegfs-test-01.log:39:PASS 228b (24s)
[centos@test-01 test_logs]$ find /tmp/test_logs/2021-10-26/034930 -name ""*suite_log* |xargs grep PASS -rin |wc -l
100
[centos@test-01 test_logs]$