[LU-13813] conf-sanity test_112: can't put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.2
• Affects Version/s: Lustre 2.14.0, Lustre 2.15.0
• Severity: 3
• Assignee: Jian Yu
• Reporter: Maloo

Description

This issue was created by maloo for vik <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/c03460d0-489d-4fbb-84e5-d7df9a59d9dc

test_112 failed with the following error:

import is not in FULL state

Investigation showed that MDS was not connected during start_mds test step in _wait_import_state()

can\'t put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN

debug log from vm2 contains messages about import state changes.
MDT0000_UUID: changing import state from DISCONN to CONNECTING and
MDT0000_UUID: changing import state from CONNECTING to DISCONN

00000100:00080000:1.0:1594563480.058266:0:25586:0:(import.c:86:import_set_state_nolock()) ffff9a285a1e5000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:1.0:1594563480.058358:0:25586:0:(pinger.c:242:ptlrpc_pinger_process_import()) 2d768d9f-aa8c-42e0-a75c-fa93c720357c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0
00000100:00080000:1.0:1594563480.058361:0:25586:0:(import.c:86:import_set_state_nolock()) ffff9a287aae0000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:1.0:1594563480.059863:0:10210:0:(import.c:86:import_set_state_nolock()) ffff9a285a1e5000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN
00000100:00080000:1.0:1594563480.059906:0:10210:0:(import.c:86:import_set_state_nolock()) ffff9a287aae0000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN
00000001:02000400:0.0:1594563481.248258:0:24754:0:(debug.c:712:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark  rpc : @@@@@@ FAIL: can\'t put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN
00000001:02000400:1.0:1594563481.249585:0:24755:0:(debug.c:712:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark  rpc : @@@@@@ FAIL: can\'t put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN

Started lustre-MDT0000
CMD: onyx-42vm1.onyx.whamcloud.com,onyx-42vm2 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config bash rpc.sh wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 
onyx-42vm1: == rpc test complete, duration -o sec ================================================================ 13:53:10 (1594561990)
onyx-42vm2: == rpc test complete, duration -o sec ================================================================ 13:53:10 (1594561990)
onyx-42vm1: CMD: onyx-42vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm2: CMD: onyx-42vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm1: CMD: onyx-42vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm2: CMD: onyx-42vm4 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm1: CMD: onyx-42vm3 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm2: CMD: onyx-42vm3 /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm1: CMD: onyx-42vm1.onyx.whamcloud.com /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm2: CMD: onyx-42vm2.onyx.whamcloud.com /usr/sbin/lctl get_param -n version 2>/dev/null
onyx-42vm1: onyx-42vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
onyx-42vm2: onyx-42vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
onyx-42vm2: CMD: onyx-42vm2.onyx.whamcloud.com lctl get_param -n at_max
onyx-42vm2:  rpc : @@@@@@ FAIL: can't put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN 
onyx-42vm2:   Trace dump:
onyx-42vm2:   = /usr/lib64/lustre/tests/test-framework.sh:6216:error()
onyx-42vm2:   = /usr/lib64/lustre/tests/test-framework.sh:7395:_wait_import_state()
onyx-42vm2:   = /usr/lib64/lustre/tests/test-framework.sh:7417:wait_import_state()
onyx-42vm2:   = /usr/lib64/lustre/tests/test-framework.sh:7426:wait_import_state_mount()
onyx-42vm2:   = rpc.sh:18:main()

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
conf-sanity test_112 - import is not in FULL state

Issue Links

• is related to: LU-13221 conf-sanity test_112: FAIL: MDS start failed
• is related to: LU-12818 replay-single test_70b and other tests fail with “Numerical result out of range” error
• is related to: LU-13184 conf-sanity test_112: problem creating f112.conf-sanity.0 on OST0000
• is related to: LU-12857 recovery-mds-scale test_failover_ost fails with “import is not in FULL state”