[LU-3372] conf-sanity test_35a: @@@@@@ FAIL: test_35a failed with 7

Thu, 9 Jan 2020 07:14:18 +0000

== conf-sanity test 35a: Reconnect to the last active server first == 21:59:11 (1369112351)
Loading modules from /usr/lib64/lustre
detected 8 online CPUs by sysfs
libcfs will create CPU partition based on online CPUs
debug=-1
subsystem_debug=0
gss/krb5 is not supported
loading modules on: 'c07,c09,mds03-ib,oss02-ib'
CMD: c07,c09,mds03-ib,oss02-ib PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin::/sbin:/bin:/usr/sbin: NAME=local sh rpc.sh load_modules_local
c09: Loading modules from /usr/lib64/lustre
c09: detected 8 online CPUs by sysfs
c09: libcfs will create CPU partition based on online CPUs
c07: Loading modules from /usr/lib64/lustre
c07: detected 8 online CPUs by sysfs
c07: libcfs will create CPU partition based on online CPUs
oss02-ib: Loading modules from /usr/lib64/lustre
oss02-ib: detected 16 online CPUs by sysfs
oss02-ib: libcfs will create CPU partition based on online CPUs
mds03-ib: Loading modules from /usr/lib64/lustre
mds03-ib: detected 16 online CPUs by sysfs
mds03-ib: libcfs will create CPU partition based on online CPUs
c09: debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
c09: subsystem_debug=all -lnet -lnd -pinger
c07: debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
c07: subsystem_debug=all -lnet -lnd -pinger
oss02-ib: debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
oss02-ib: subsystem_debug=all -lnet -lnd -pinger
mds03-ib: debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super
mds03-ib: subsystem_debug=all -lnet -lnd -pinger
c07: gss/krb5 is not supported
c09: gss/krb5 is not supported
oss02-ib: gss/krb5 is not supported
mds03-ib: gss/krb5 is not supported
oss02-ib: quota/lquota options: 'hash_lqs_cur_bits=3'
mds03-ib: quota/lquota options: 'hash_lqs_cur_bits=3'
start mds service on mds03-ib
CMD: mds03-ib mkdir -p /mnt/mds1
CMD: mds03-ib test -b /dev/sdb
Starting mds1: /dev/sdb /mnt/mds1
CMD: mds03-ib mkdir -p /mnt/mds1; mount -t lustre /dev/sdb /mnt/mds1
CMD: mds03-ib PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin::/sbin:/bin:/usr/sbin: NAME=local sh rpc.sh set_default_debug \"-1\" \"0\" 48
CMD: mds03-ib e2label /dev/sdb 2>/dev/null
Started smplust-MDT0000
start ost1 service on oss02-ib
CMD: oss02-ib mkdir -p /mnt/ost1
CMD: oss02-ib test -b /dev/sdc
Starting ost1: /dev/sdc /mnt/ost1
CMD: oss02-ib mkdir -p /mnt/ost1; mount -t lustre /dev/sdc /mnt/ost1
CMD: oss02-ib PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin::/sbin:/bin:/usr/sbin: NAME=local sh rpc.sh set_default_debug \"-1\" \"0\" 48
CMD: oss02-ib e2label /dev/sdc 2>/dev/null
Started smplust-OST0000
mount smplust on /mnt/lustre.....
Starting client: c08: -o user_xattr,flock mds03-ib@o2ib:/smplust /mnt/lustre
CMD: c08 mkdir -p /mnt/lustre
CMD: c08 mount -t lustre -o user_xattr,flock mds03-ib@o2ib:/smplust /mnt/lustre
debug=ha
Set up a fake failnode for the MDS
CMD: mds03-ib lctl get_param -n devices
CMD: mds03-ib /usr/sbin/lctl conf_param smplust-MDT0000.failover.node= 127.0.0.2@o2ib
Wait for RECONNECT_INTERVAL seconds (10s)
conf-sanity.sh test_35a 2013-05-2021h59m53s
Stopping the MDT: smplust-MDT0000
stop mds service on mds03-ib
CMD: mds03-ib grep -c /mnt/mds1' ' /proc/mounts
Stopping /mnt/mds1 (opts:-f) on mds03-ib
CMD: mds03-ib umount -d -f /mnt/mds1
CMD: mds03-ib lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '
Restarting the MDT: smplust-MDT0000
start mds service on mds03-ib
CMD: mds03-ib mkdir -p /mnt/mds1
CMD: mds03-ib test -b /dev/sdb
Starting mds1: /dev/sdb /mnt/mds1
CMD: mds03-ib mkdir -p /mnt/mds1; mount -t lustre /dev/sdb /mnt/mds1
CMD: mds03-ib PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin::/sbin:/bin:/usr/sbin: NAME=local sh rpc.sh set_default_debug \"-1\" \"0\" 48
CMD: mds03-ib e2label /dev/sdb 2>/dev/null
Started smplust-MDT0000
Wait for df (8554) ...
done
debug=trace inode super ext2 malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck
Debug log: 21 lines, 21 kept, 0 dropped, 0 bad.
The client didn't try to reconnect to the last active server (tried instead)
conf-sanity test_35a: @@@@@@ FAIL: test_35a failed with 7
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:4186:error_noexit()
= /usr/lib64/lustre/tests/test-framework.sh:4213:error()
= /usr/lib64/lustre/tests/test-framework.sh:4452:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:4485:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:4307:run_test()
= /usr/lib64/lustre/tests/conf-sanity.sh:2042:main()
Dumping lctl log to /scratch/tmp/minh/logs//2013-05-20/215628/conf-sanity.test_35a.*.1369112420.log
CMD: c07,c08,c09,mds03-ib,oss02-ib /usr/sbin/lctl dk > /scratch/tmp/minh/logs//2013-05-20/215628/conf-sanity.test_35a.debug_log.\$(hostname -s).1369112420.log;
dmesg > /scratch/tmp/minh/logs//2013-05-20/215628/conf-sanity.test_35a.dmesg.\$(hostname -s).1369112420.log
CMD: c07,c08,c09,mds03-ib,oss02-ib rsync -az /scratch/tmp/minh/logs//2013-05-20/215628/conf-sanity.test_35a.*.1369112420.log c08:/scratch/tmp/minh/logs//2013-05-20/215628

Whamcloud Community JIRA

[LU-3372] conf-sanity test_35a: @@@@@@ FAIL: test_35a failed with 7