[LU-7165] Failover: replay-ost-single test_0b: import is not in FULL state

Tue, 14 Dec 2021 22:22:10 +0000

This issue was created by maloo for Saurabh Tandan <saurabh.tandan@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/46803cda-52de-11e5-b61c-5254006e85c2.

The sub-test test_0b failed with the following error:

import is not in FULL state

Test log:

Started lustre:OST0006
CMD: shadow-16vm5,shadow-16vm6,shadow-16vm9 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/mpi/gcc/openmpi/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh wait_import_state_mount FULL osc.lustre:OST0000-osc-*.ost_server_uuid,osc.lustre:OST0001-osc-*.ost_server_uuid,osc.lustre:OST0002-osc-*.ost_server_uuid,osc.lustre:OST0003-osc-*.ost_server_uuid,osc.lustre:OST0004-osc-*.ost_server_uuid,osc.lustre:OST0005-osc-*.ost_server_uuid,osc.lustre:OST0006-osc-*.ost_server_uuid 
shadow-16vm9: CMD: shadow-16vm9 lctl get_param -n at_max
shadow-16vm5: CMD: shadow-16vm5 lctl get_param -n at_max
shadow-16vm6: CMD: shadow-16vm6 lctl get_param -n at_max
shadow-16vm9:  rpc : @@@@@@ FAIL: can't put import for osc.lustre:OST0000-osc-*.ost_server_uuid into FULL state after 1475 sec, have  
shadow-16vm6:  rpc : @@@@@@ FAIL: can't put import for osc.lustre:OST0000-osc-*.ost_server_uuid into FULL state after 1475 sec, have  
shadow-16vm5:  rpc : @@@@@@ FAIL: can't put import for osc.lustre:OST0000-osc-*.ost_server_uuid into FULL state after 1475 sec, have  
shadow-16vm9:   Trace dump:
shadow-16vm9:   = /usr/lib64/lustre/tests/test-framework.sh:4748:error_noexit()
shadow-16vm9:   = /usr/lib64/lustre/tests/test-framework.sh:4779:error()
shadow-16vm9:   = /usr/lib64/lustre/tests/test-framework.sh:5837:_wait_import_state()
shadow-16vm9:   = /usr/lib64/lustre/tests/test-framework.sh:5856:wait_import_state()
shadow-16vm9:   = /usr/lib64/lustre/tests/test-framework.sh:5865:wait_import_state_mount()
shadow-16vm9:   = rpc.sh:20:main()
shadow-16vm9: CMD: shadow-16vm3,shadow-16vm5,shadow-16vm6,shadow-16vm8,shadow-16vm9 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/mpi/gcc/openmpi/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_logdir /tmp/test_logs/1441280803 
shadow-16vm9: CMD: shadow-16vm3 uname -n
shadow-16vm9: Dumping lctl log to /tmp/test_logs/1441280803/rpc..*.1441282291.log
shadow-16vm9: CMD: shadow-16vm3,shadow-16vm5,shadow-16vm6,shadow-16vm8,shadow-16vm9 /usr/sbin/lctl dk > /tmp/test_logs/1441280803/rpc..debug_log.\$(hostname -s).1441282291.log;
shadow-16vm9:          dmesg > /tmp/test_logs/1441280803/rpc..dmesg.\$(hostname -s).1441282291.log
shadow-16vm9: CMD: shadow-16vm3,shadow-16vm5,shadow-16vm6,shadow-16vm8,shadow-16vm9 rsync -az /tmp/test_logs/1441280803/rpc..*.1441282291.log shadow-16vm9:/tmp/test_logs/1441280803
shadow-16vm5:   Trace dump:
shadow-16vm5:   = /usr/lib64/lustre/tests/test-framework.sh:4748:error_noexit()
shadow-16vm5:   = /usr/lib64/lustre/tests/test-framework.sh:4779:error()
shadow-16vm5:   = /usr/lib64/lustre/tests/test-framework.sh:5837:_wait_import_state()
shadow-16vm5:   = /usr/lib64/lustre/tests/test-framework.sh:5856:wait_import_state()
shadow-16vm5:   = /usr/lib64/lustre/tests/test-framework.sh:5865:wait_import_state_mount()
shadow-16vm5:   = rpc.sh:20:main()
shadow-16vm6:   Trace dump:
shadow-16vm6:   = /usr/lib64/lustre/tests/test-framework.sh:4748:error_noexit()
shadow-16vm6:   = /usr/lib64/lustre/tests/test-framework.sh:4779:error()
shadow-16vm6:   = /usr/lib64/lustre/tests/test-framework.sh:5837:_wait_import_state()
shadow-16vm6:   = /usr/lib64/lustre/tests/test-framework.sh:5856:wait_import_state()
shadow-16vm6:   = /usr/lib64/lustre/tests/test-framework.sh:5865:wait_import_state_mount()
shadow-16vm6:   = rpc.sh:20:main()
shadow-16vm5: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/mpi/gcc/openmpi/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_logdir /tmp/test_logs/1441280803 
shadow-16vm6: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/mpi/gcc/openmpi/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_logdir /tmp/test_logs/1441280803 
shadow-16vm5: CMD: shadow-16vm5 uname -n
shadow-16vm6: CMD: shadow-16vm5 uname -n
shadow-16vm5: CMD: shadow-16vm6 uname -n
shadow-16vm5: Dumping lctl log to /tmp/test_logs/1441280803/rpc..*.1441282293.log
shadow-16vm5: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 /usr/sbin/lctl dk > /tmp/test_logs/1441280803/rpc..debug_log.\$(hostname -s).1441282293.log;
shadow-16vm5:          dmesg > /tmp/test_logs/1441280803/rpc..dmesg.\$(hostname -s).1441282293.log
shadow-16vm6: Dumping lctl log to /tmp/test_logs/1441280803/rpc..*.1441282293.log
shadow-16vm6: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 /usr/sbin/lctl dk > /tmp/test_logs/1441280803/rpc..debug_log.\$(hostname -s).1441282293.log;
shadow-16vm6:          dmesg > /tmp/test_logs/1441280803/rpc..dmesg.\$(hostname -s).1441282293.log
shadow-16vm5: shadow-16vm7: open /proc/sys/lnet/dump_kernel failed: No such file or directory
shadow-16vm5: shadow-16vm7: open(dump_kernel) failed: No such file or directory
shadow-16vm5: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 rsync -az /tmp/test_logs/1441280803/rpc..*.1441282293.log shadow-16vm5:/tmp/test_logs/1441280803
shadow-16vm6: shadow-16vm7: open /proc/sys/lnet/dump_kernel failed: No such file or directory
shadow-16vm6: shadow-16vm7: open(dump_kernel) failed: No such file or directory
shadow-16vm6: CMD: shadow-16vm5,shadow-16vm6,shadow-16vm7,shadow-16vm8 rsync -az /tmp/test_logs/1441280803/rpc..*.1441282293.log shadow-16vm6:/tmp/test_logs/1441280803
shadow-16vm5: shadow-16vm6: Warning: Permanently added 'shadow-16vm5,10.1.4.189' (ECDSA) to the list of known hosts.
shadow-16vm5: shadow-16vm5: Warning: Permanently added 'shadow-16vm5,10.1.4.189' (ECDSA) to the list of known hosts.
shadow-16vm6: shadow-16vm6: Warning: Permanently added 'shadow-16vm6,10.1.4.190' (ECDSA) to the list of known hosts.
shadow-16vm6: shadow-16vm5: Warning: Permanently added 'shadow-16vm6,10.1.4.190' (ECDSA) to the list of known hosts.
 replay-ost-single test_0b: @@@@@@ FAIL: import is not in FULL state

LustreError inside dmesg for replay-ost-single test 0b in mds :

[ 3574.906399] LustreError: 16331:0:(events.c:306:request_in_callback()) event type 2, status -5, service mdt

Whamcloud Community JIRA

[LU-7165] Failover: replay-ost-single test_0b: import is not in FULL state