Sep 4 18:03:51 fat-intel-3vm8 xinetd[1776]: START: shell pid=1890 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[1890]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[1890]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[1891]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "hostname";echo XXRETCODE:$?' Sep 4 18:03:52 fat-intel-3vm8 rshd[1890]: pam_unix(rsh:session): session closed for user root Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=1890 duration=1(sec) Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=1915 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[1915]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[1915]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[1916]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "hostname");echo XXRETCODE:$?' Sep 4 18:03:52 fat-intel-3vm8 rshd[1915]: pam_unix(rsh:session): session closed for user root Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=1915 duration=0(sec) Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=1939 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[1939]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[1939]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[1940]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P1";echo XXRETCODE:$?' Sep 4 18:03:52 fat-intel-3vm8 rshd[1939]: pam_unix(rsh:session): session closed for user root Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=1939 duration=0(sec) Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=1964 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[1964]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[1964]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[1965]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P1");echo XXRETCODE:$?' Sep 4 18:03:52 fat-intel-3vm8 rshd[1964]: pam_unix(rsh:session): session closed for user root Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=1964 duration=0(sec) Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=1988 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[1988]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[1988]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[1989]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost1; mount -t lustre #011#011 /dev/lvm-OSS/P1 /mnt/ost1";echo XXRETCODE:$?' Sep 4 18:03:52 fat-intel-3vm8 rshd[1988]: pam_unix(rsh:session): session closed for user root Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=1988 duration=0(sec) Sep 4 18:03:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=2013 from=::ffff:10.10.4.86 Sep 4 18:03:52 fat-intel-3vm8 rshd[2013]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:03:52 fat-intel-3vm8 rshd[2013]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:03:52 fat-intel-3vm8 rshd[2014]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost1; mount -t lustre #011#011 /dev/lvm-OSS/P1 /mnt/ost1");echo XXRETCODE:$?' Sep 4 18:03:53 fat-intel-3vm8 kernel: LNet: HW CPU cores: 1, npartitions: 1 Sep 4 18:03:53 fat-intel-3vm8 kernel: alg: No test for crc32 (crc32-table) Sep 4 18:03:53 fat-intel-3vm8 kernel: alg: No test for adler32 (adler32-zlib) Sep 4 18:03:57 fat-intel-3vm8 kernel: padlock: VIA PadLock Hash Engine not detected. Sep 4 18:03:57 fat-intel-3vm8 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-279.5.1.el6_lustre.ga24ad33.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device Sep 4 18:04:01 fat-intel-3vm8 kernel: Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel-12-g762d23d-PRISTINE-2.6.32-279.5.1.el6_lustre.ga24ad33.x86_64 Sep 4 18:04:02 fat-intel-3vm8 kernel: LNet: Added LNI 10.10.4.93@tcp [8/256/0/180] Sep 4 18:04:02 fat-intel-3vm8 kernel: LNet: Accept all, port 7988 Sep 4 18:04:03 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:04:03 fat-intel-3vm8 kernel: Sep 4 18:04:45 fat-intel-3vm8 kernel: LDISKFS-fs (dm-0): recovery complete Sep 4 18:04:45 fat-intel-3vm8 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:04:45 fat-intel-3vm8 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:04:50 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807085/real 1346807085] req@ffff88007fac9800 x1412229540872193/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807090 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:04:56 fat-intel-3vm8 kernel: LustreError: 2039:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880072076000 x1412229540872194/t0(0) o253->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:04:56 fat-intel-3vm8 kernel: LustreError: 2039:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:05:02 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0006_UUID' is not available for connect (no target) Sep 4 18:05:02 fat-intel-3vm8 kernel: LustreError: 2039:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880072076000 x1412229540872195/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:05:05 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0000_UUID' is not available for connect (no target) Sep 4 18:05:06 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (no target) Sep 4 18:05:06 fat-intel-3vm8 kernel: LustreError: Skipped 5 previous similar messages Sep 4 18:05:08 fat-intel-3vm8 kernel: LustreError: 2039:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880072076000 x1412229540872196/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:05:08 fat-intel-3vm8 kernel: Lustre: 2123:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0000, 4 recoverable clients, last_transno 6210 Sep 4 18:05:08 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/dm-0 with recovery enabled Sep 4 18:05:15 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (no target) Sep 4 18:05:15 fat-intel-3vm8 kernel: LustreError: Skipped 5 previous similar messages Sep 4 18:05:18 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (no target) Sep 4 18:05:18 fat-intel-3vm8 kernel: LustreError: Skipped 5 previous similar messages Sep 4 18:05:19 fat-intel-3vm8 kernel: LustreError: 2039:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880072076000 x1412229540872197/t0(0) o253->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:05:19 fat-intel-3vm8 rshd[2013]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:19 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2013 duration=87(sec) Sep 4 18:05:19 fat-intel-3vm8 xinetd[1776]: START: shell pid=2131 from=::ffff:10.10.4.86 Sep 4 18:05:19 fat-intel-3vm8 rshd[2131]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2131]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2132]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:05:20 fat-intel-3vm8 rshd[2131]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2131 duration=1(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2154 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2154]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2154]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2155]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: LNet: 2260:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:05:20 fat-intel-3vm8 rshd[2154]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2154 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2263 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2263]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2263]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2264]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P1 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P1 2>/dev/null Sep 4 18:05:20 fat-intel-3vm8 rshd[2263]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2263 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2286 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2286]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2286]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2287]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P1 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 rshd[2286]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2286 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2311 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2311]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2311]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2312]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P2";echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P2 Sep 4 18:05:20 fat-intel-3vm8 rshd[2311]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2311 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2334 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807110/real 1346807110] req@ffff88007215f400 x1412229540872198/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807120 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:05:20 fat-intel-3vm8 rshd[2334]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2334]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2335]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P2");echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 rshd[2334]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2334 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2358 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2358]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2358]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2359]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost2; mount -t lustre #011#011 /dev/lvm-OSS/P2 /mnt/ost2";echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost2; mount -t lustre #011#011 /dev/lvm-OSS/P2 /mnt/ost2 Sep 4 18:05:20 fat-intel-3vm8 rshd[2358]: pam_unix(rsh:session): session closed for user root Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2358 duration=0(sec) Sep 4 18:05:20 fat-intel-3vm8 xinetd[1776]: START: shell pid=2381 from=::ffff:10.10.4.86 Sep 4 18:05:20 fat-intel-3vm8 rshd[2381]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:05:20 fat-intel-3vm8 rshd[2381]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:05:20 fat-intel-3vm8 rshd[2382]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost2; mount -t lustre #011#011 /dev/lvm-OSS/P2 /mnt/ost2");echo XXRETCODE:$?' Sep 4 18:05:20 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:05:20 fat-intel-3vm8 kernel: Sep 4 18:05:22 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:05:25 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0001_UUID' is not available for connect (no target) Sep 4 18:05:25 fat-intel-3vm8 kernel: LustreError: Skipped 5 previous similar messages Sep 4 18:05:35 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0002_UUID' is not available for connect (no target) Sep 4 18:05:35 fat-intel-3vm8 kernel: LustreError: Skipped 6 previous similar messages Sep 4 18:05:40 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807125/real 1346807125] req@ffff88006fd50400 x1412229540872199/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807140 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:05:52 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0006_UUID' is not available for connect (no target) Sep 4 18:05:52 fat-intel-3vm8 kernel: LustreError: Skipped 24 previous similar messages Sep 4 18:06:03 fat-intel-3vm8 kernel: LDISKFS-fs (dm-1): recovery complete Sep 4 18:06:03 fat-intel-3vm8 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:06:03 fat-intel-3vm8 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:06:03 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:06:16 fat-intel-3vm8 ntpd[1784]: synchronized to 10.10.0.1, stratum 2 Sep 4 18:06:16 fat-intel-3vm8 ntpd[1784]: time reset +1.935346 s Sep 4 18:06:16 fat-intel-3vm8 ntpd[1784]: kernel time sync status change 2001 Sep 4 18:06:21 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880037d39400 x1412229540872202/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:06:24 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: recovery is timed out, evict stale exports Sep 4 18:06:24 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: disconnecting 1 stale clients Sep 4 18:06:24 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: Recovery over after 1:02, of 4 clients 3 recovered and 1 was evicted. Sep 4 18:06:24 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: received MDS connection from 10.10.4.92@tcp Sep 4 18:06:27 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880037d39400 x1412229540872203/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:06:27 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0002_UUID' is not available for connect (no target) Sep 4 18:06:27 fat-intel-3vm8 kernel: LustreError: Skipped 42 previous similar messages Sep 4 18:06:28 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807163/real 1346807163] req@ffff880072797c00 x1412229540872201/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807188 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:06:35 fat-intel-3vm8 kernel: LustreError: 2407:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880037d39000 x1412229540872245/t0(0) o253->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:06:35 fat-intel-3vm8 kernel: LustreError: 2407:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:06:41 fat-intel-3vm8 kernel: Lustre: 2435:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0001, 4 recoverable clients, last_transno 5948 Sep 4 18:06:41 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Now serving lustre-OST0001 on /dev/dm-1 with recovery enabled Sep 4 18:06:47 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:06:48 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Temporarily refusing client connection from 10.10.4.86@tcp Sep 4 18:06:52 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:06:52 fat-intel-3vm8 kernel: Lustre: Skipped 1 previous similar message Sep 4 18:06:52 fat-intel-3vm8 kernel: LustreError: 2407:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880037d39000 x1412229540872248/t0(0) o253->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:06:52 fat-intel-3vm8 kernel: LustreError: 2407:0:(client.c:1114:ptlrpc_import_delay_req()) Skipped 1 previous similar message Sep 4 18:06:52 fat-intel-3vm8 rshd[2381]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:52 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2381 duration=92(sec) Sep 4 18:06:52 fat-intel-3vm8 xinetd[1776]: START: shell pid=2438 from=::ffff:10.10.4.86 Sep 4 18:06:52 fat-intel-3vm8 rshd[2438]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:52 fat-intel-3vm8 rshd[2438]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:52 fat-intel-3vm8 rshd[2439]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:06:53 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:06:53 fat-intel-3vm8 rshd[2438]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2438 duration=1(sec) Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: START: shell pid=2461 from=::ffff:10.10.4.86 Sep 4 18:06:53 fat-intel-3vm8 rshd[2461]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:53 fat-intel-3vm8 rshd[2461]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:53 fat-intel-3vm8 rshd[2462]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:06:53 fat-intel-3vm8 kernel: LNet: 2567:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:06:53 fat-intel-3vm8 rshd[2461]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2461 duration=0(sec) Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: START: shell pid=2570 from=::ffff:10.10.4.86 Sep 4 18:06:53 fat-intel-3vm8 rshd[2570]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:53 fat-intel-3vm8 rshd[2570]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:53 fat-intel-3vm8 rshd[2571]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P2 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:06:53 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P2 2>/dev/null Sep 4 18:06:53 fat-intel-3vm8 rshd[2570]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2570 duration=0(sec) Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: START: shell pid=2593 from=::ffff:10.10.4.86 Sep 4 18:06:53 fat-intel-3vm8 rshd[2593]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:53 fat-intel-3vm8 rshd[2593]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:53 fat-intel-3vm8 rshd[2594]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P2 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:06:53 fat-intel-3vm8 rshd[2593]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2593 duration=0(sec) Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: START: shell pid=2618 from=::ffff:10.10.4.86 Sep 4 18:06:53 fat-intel-3vm8 rshd[2618]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:53 fat-intel-3vm8 rshd[2618]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:53 fat-intel-3vm8 rshd[2619]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P3";echo XXRETCODE:$?' Sep 4 18:06:53 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P3 Sep 4 18:06:53 fat-intel-3vm8 rshd[2618]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2618 duration=0(sec) Sep 4 18:06:53 fat-intel-3vm8 xinetd[1776]: START: shell pid=2641 from=::ffff:10.10.4.86 Sep 4 18:06:53 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:06:57 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807192/real 1346807192] req@ffff88007161b000 x1412229540872246/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807217 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:06:58 fat-intel-3vm8 rshd[2641]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:58 fat-intel-3vm8 rshd[2641]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:58 fat-intel-3vm8 rshd[2643]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P3");echo XXRETCODE:$?' Sep 4 18:06:58 fat-intel-3vm8 rshd[2641]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:58 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2641 duration=5(sec) Sep 4 18:06:58 fat-intel-3vm8 xinetd[1776]: START: shell pid=2666 from=::ffff:10.10.4.86 Sep 4 18:06:58 fat-intel-3vm8 rshd[2666]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:58 fat-intel-3vm8 rshd[2666]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:58 fat-intel-3vm8 rshd[2667]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost3; mount -t lustre #011#011 /dev/lvm-OSS/P3 /mnt/ost3";echo XXRETCODE:$?' Sep 4 18:06:58 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost3; mount -t lustre #011#011 /dev/lvm-OSS/P3 /mnt/ost3 Sep 4 18:06:58 fat-intel-3vm8 rshd[2666]: pam_unix(rsh:session): session closed for user root Sep 4 18:06:58 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2666 duration=0(sec) Sep 4 18:06:58 fat-intel-3vm8 xinetd[1776]: START: shell pid=2689 from=::ffff:10.10.4.86 Sep 4 18:06:58 fat-intel-3vm8 rshd[2689]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:06:58 fat-intel-3vm8 rshd[2689]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:06:58 fat-intel-3vm8 rshd[2690]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost3; mount -t lustre #011#011 /dev/lvm-OSS/P3 /mnt/ost3");echo XXRETCODE:$?' Sep 4 18:06:58 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:06:58 fat-intel-3vm8 kernel: Sep 4 18:07:00 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: Recovery over after 0:07, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:07:00 fat-intel-3vm8 kernel: Lustre: lustre-OST0001: received MDS connection from 10.10.4.92@tcp Sep 4 18:07:27 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807222/real 1346807222] req@ffff880072784000 x1412229540872290/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807247 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:07:38 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0005_UUID' is not available for connect (no target) Sep 4 18:07:38 fat-intel-3vm8 kernel: LustreError: Skipped 92 previous similar messages Sep 4 18:07:41 fat-intel-3vm8 kernel: LDISKFS-fs (dm-2): recovery complete Sep 4 18:07:41 fat-intel-3vm8 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:07:41 fat-intel-3vm8 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:07:41 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:07:56 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880037d03800 x1412229540872293/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:08:06 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807261/real 1346807261] req@ffff8800712dfc00 x1412229540872292/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807286 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:08:11 fat-intel-3vm8 kernel: LustreError: 2715:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:08:17 fat-intel-3vm8 kernel: Lustre: 2743:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0002, 4 recoverable clients, last_transno 5945 Sep 4 18:08:17 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Now serving lustre-OST0002 on /dev/dm-2 with recovery enabled Sep 4 18:08:18 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Temporarily refusing client connection from 10.10.4.86@tcp Sep 4 18:08:22 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:08:27 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:08:27 fat-intel-3vm8 kernel: Lustre: Skipped 2 previous similar messages Sep 4 18:08:28 fat-intel-3vm8 rshd[2689]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:28 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2689 duration=90(sec) Sep 4 18:08:28 fat-intel-3vm8 xinetd[1776]: START: shell pid=2747 from=::ffff:10.10.4.86 Sep 4 18:08:28 fat-intel-3vm8 rshd[2747]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:28 fat-intel-3vm8 rshd[2747]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:28 fat-intel-3vm8 rshd[2748]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:08:29 fat-intel-3vm8 rshd[2747]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2747 duration=1(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2770 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2770]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2770]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2771]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 kernel: LNet: 2876:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:08:29 fat-intel-3vm8 rshd[2770]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2770 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2879 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2879]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2879]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2880]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P3 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P3 2>/dev/null Sep 4 18:08:29 fat-intel-3vm8 rshd[2879]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2879 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2902 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2902]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2902]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2903]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P3 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 rshd[2902]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2902 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2927 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2927]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2927]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2928]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P4";echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 rshd[2927]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P4 Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2927 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2950 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2950]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2950]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2951]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P4");echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 rshd[2950]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2950 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2974 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2974]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2974]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2975]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost4; mount -t lustre #011#011 /dev/lvm-OSS/P4 /mnt/ost4";echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost4; mount -t lustre #011#011 /dev/lvm-OSS/P4 /mnt/ost4 Sep 4 18:08:29 fat-intel-3vm8 rshd[2974]: pam_unix(rsh:session): session closed for user root Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2974 duration=0(sec) Sep 4 18:08:29 fat-intel-3vm8 xinetd[1776]: START: shell pid=2997 from=::ffff:10.10.4.86 Sep 4 18:08:29 fat-intel-3vm8 rshd[2997]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:08:29 fat-intel-3vm8 rshd[2997]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:08:29 fat-intel-3vm8 rshd[2998]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost4; mount -t lustre #011#011 /dev/lvm-OSS/P4 /mnt/ost4");echo XXRETCODE:$?' Sep 4 18:08:29 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-3): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:08:29 fat-intel-3vm8 kernel: Sep 4 18:08:30 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:08:32 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807287/real 1346807287] req@ffff8800704cac00 x1412229540872297/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807312 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:08:44 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: Recovery over after 0:14, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:08:44 fat-intel-3vm8 kernel: Lustre: lustre-OST0002: received MDS connection from 10.10.4.92@tcp Sep 4 18:09:12 fat-intel-3vm8 kernel: LDISKFS-fs (dm-3): recovery complete Sep 4 18:09:12 fat-intel-3vm8 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:09:13 fat-intel-3vm8 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:09:13 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:09:25 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff88007598a000 x1412229540872344/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:09:25 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Sep 4 18:09:38 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807353/real 1346807353] req@ffff880075988c00 x1412229540872343/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807378 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:09:38 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 4 18:09:43 fat-intel-3vm8 kernel: LustreError: 3023:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:09:47 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0003_UUID' is not available for connect (no target) Sep 4 18:09:47 fat-intel-3vm8 kernel: LustreError: Skipped 147 previous similar messages Sep 4 18:09:49 fat-intel-3vm8 kernel: Lustre: 3051:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0003, 4 recoverable clients, last_transno 5940 Sep 4 18:09:49 fat-intel-3vm8 kernel: Lustre: lustre-OST0003: Now serving lustre-OST0003 on /dev/dm-3 with recovery enabled Sep 4 18:09:57 fat-intel-3vm8 kernel: Lustre: lustre-OST0003: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:09:57 fat-intel-3vm8 kernel: Lustre: Skipped 1 previous similar message Sep 4 18:10:00 fat-intel-3vm8 rshd[2997]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=2997 duration=91(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3055 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3055]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3055]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3056]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:10:00 fat-intel-3vm8 rshd[3055]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3055 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3078 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3078]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3078]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3079]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 kernel: LNet: 3184:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:10:00 fat-intel-3vm8 rshd[3078]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3078 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3187 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3187]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3187]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3188]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P4 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P4 2>/dev/null Sep 4 18:10:00 fat-intel-3vm8 rshd[3187]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3187 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3210 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3210]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3210]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3211]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P4 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 rshd[3210]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3210 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3235 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3235]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3235]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3236]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P5";echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P5 Sep 4 18:10:00 fat-intel-3vm8 rshd[3235]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3235 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3258 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3258]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3258]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3259]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P5");echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 rshd[3258]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3258 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3282 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3282]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3282]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3283]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost5; mount -t lustre #011#011 /dev/lvm-OSS/P5 /mnt/ost5";echo XXRETCODE:$?' Sep 4 18:10:00 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost5; mount -t lustre #011#011 /dev/lvm-OSS/P5 /mnt/ost5 Sep 4 18:10:00 fat-intel-3vm8 rshd[3282]: pam_unix(rsh:session): session closed for user root Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3282 duration=0(sec) Sep 4 18:10:00 fat-intel-3vm8 xinetd[1776]: START: shell pid=3305 from=::ffff:10.10.4.86 Sep 4 18:10:00 fat-intel-3vm8 rshd[3305]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:10:00 fat-intel-3vm8 rshd[3305]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:10:00 fat-intel-3vm8 rshd[3306]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost5; mount -t lustre #011#011 /dev/lvm-OSS/P5 /mnt/ost5");echo XXRETCODE:$?' Sep 4 18:10:01 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-4): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:10:01 fat-intel-3vm8 kernel: Sep 4 18:10:01 fat-intel-3vm8 CROND[3336]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 18:10:02 fat-intel-3vm8 kernel: Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:10:10 fat-intel-3vm8 kernel: Lustre: lustre-OST0003: Recovery over after 0:08, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:10:10 fat-intel-3vm8 kernel: Lustre: lustre-OST0003: received MDS connection from 10.10.4.92@tcp Sep 4 18:10:43 fat-intel-3vm8 kernel: LDISKFS-fs (dm-4): recovery complete Sep 4 18:10:43 fat-intel-3vm8 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:10:44 fat-intel-3vm8 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:10:44 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:11:09 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807444/real 1346807444] req@ffff8800712ae800 x1412229540872392/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:11:09 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 4 18:11:14 fat-intel-3vm8 kernel: LustreError: 3331:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:11:20 fat-intel-3vm8 kernel: Lustre: 3362:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0004, 4 recoverable clients, last_transno 5945 Sep 4 18:11:20 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: Now serving lustre-OST0004 on /dev/dm-4 with recovery enabled Sep 4 18:11:22 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:11:22 fat-intel-3vm8 kernel: Lustre: Skipped 1 previous similar message Sep 4 18:11:31 fat-intel-3vm8 rshd[3305]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3305 duration=91(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3365 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3365]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3365]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3366]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:11:31 fat-intel-3vm8 rshd[3365]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3365 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3388 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3388]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3388]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3389]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: LNet: 3494:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:11:31 fat-intel-3vm8 rshd[3388]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3388 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3497 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3497]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3497]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3498]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P5 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P5 2>/dev/null Sep 4 18:11:31 fat-intel-3vm8 rshd[3497]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3520 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3497 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 rshd[3520]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3520]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3521]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P5 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 rshd[3520]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3520 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3545 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3545]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3545]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3546]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P6";echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P6 Sep 4 18:11:31 fat-intel-3vm8 rshd[3545]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3545 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3568 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3568]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3568]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3569]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P6");echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 rshd[3568]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3568 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3592 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3592]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3592]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3593]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost6; mount -t lustre #011#011 /dev/lvm-OSS/P6 /mnt/ost6";echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost6; mount -t lustre #011#011 /dev/lvm-OSS/P6 /mnt/ost6 Sep 4 18:11:31 fat-intel-3vm8 rshd[3592]: pam_unix(rsh:session): session closed for user root Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3592 duration=0(sec) Sep 4 18:11:31 fat-intel-3vm8 xinetd[1776]: START: shell pid=3615 from=::ffff:10.10.4.86 Sep 4 18:11:31 fat-intel-3vm8 rshd[3615]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:11:31 fat-intel-3vm8 rshd[3615]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:11:31 fat-intel-3vm8 rshd[3616]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost6; mount -t lustre #011#011 /dev/lvm-OSS/P6 /mnt/ost6");echo XXRETCODE:$?' Sep 4 18:11:31 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-5): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:11:31 fat-intel-3vm8 kernel: Sep 4 18:11:32 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:11:35 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: Recovery over after 0:03, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:11:35 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: received MDS connection from 10.10.4.92@tcp Sep 4 18:12:14 fat-intel-3vm8 kernel: LDISKFS-fs (dm-5): recovery complete Sep 4 18:12:14 fat-intel-3vm8 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:12:15 fat-intel-3vm8 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:12:15 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:12:31 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880075988000 x1412229540872443/t0(0) o101->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 328/384 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 Sep 4 18:12:31 fat-intel-3vm8 kernel: LustreError: 2107:0:(client.c:1114:ptlrpc_import_delay_req()) Skipped 14 previous similar messages Sep 4 18:12:46 fat-intel-3vm8 kernel: LustreError: 3641:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:12:52 fat-intel-3vm8 kernel: Lustre: 3669:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0005, 4 recoverable clients, last_transno 5946 Sep 4 18:12:52 fat-intel-3vm8 kernel: Lustre: lustre-OST0005: Now serving lustre-OST0005 on /dev/dm-5 with recovery enabled Sep 4 18:13:02 fat-intel-3vm8 kernel: Lustre: lustre-OST0005: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:13:02 fat-intel-3vm8 kernel: Lustre: Skipped 7 previous similar messages Sep 4 18:13:03 fat-intel-3vm8 rshd[3615]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3615 duration=92(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3672 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3672]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3672]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3673]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:13:03 fat-intel-3vm8 rshd[3672]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3672 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3695 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3695]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3695]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3696]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 kernel: LNet: 3801:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:13:03 fat-intel-3vm8 rshd[3695]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3695 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3804 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3804]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3804]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3805]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P6 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P6 2>/dev/null Sep 4 18:13:03 fat-intel-3vm8 rshd[3804]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3804 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3827 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3827]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3827]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3828]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P6 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 rshd[3827]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3827 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3852 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3852]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3852]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3853]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "test -b /dev/lvm-OSS/P7";echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: test -b /dev/lvm-OSS/P7 Sep 4 18:13:03 fat-intel-3vm8 rshd[3852]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3852 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3875 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3875]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3875]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3876]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "test -b /dev/lvm-OSS/P7");echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 rshd[3875]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3875 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3899 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3899]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3899]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3900]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "mkdir -p /mnt/ost7; mount -t lustre #011#011 /dev/lvm-OSS/P7 /mnt/ost7";echo XXRETCODE:$?' Sep 4 18:13:03 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: mkdir -p /mnt/ost7; mount -t lustre #011#011 /dev/lvm-OSS/P7 /mnt/ost7 Sep 4 18:13:03 fat-intel-3vm8 rshd[3899]: pam_unix(rsh:session): session closed for user root Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3899 duration=0(sec) Sep 4 18:13:03 fat-intel-3vm8 xinetd[1776]: START: shell pid=3922 from=::ffff:10.10.4.86 Sep 4 18:13:03 fat-intel-3vm8 rshd[3922]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:13:03 fat-intel-3vm8 rshd[3922]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:13:03 fat-intel-3vm8 rshd[3923]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/ost7; mount -t lustre #011#011 /dev/lvm-OSS/P7 /mnt/ost7");echo XXRETCODE:$?' Sep 4 18:13:04 fat-intel-3vm8 kernel: LDISKFS-fs warning (device dm-6): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait. Sep 4 18:13:04 fat-intel-3vm8 kernel: Sep 4 18:13:04 fat-intel-3vm8 kernel: Lustre: lustre-OST0005: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:13:25 fat-intel-3vm8 kernel: Lustre: lustre-OST0005: Recovery over after 0:21, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:13:25 fat-intel-3vm8 kernel: Lustre: lustre-OST0005: received MDS connection from 10.10.4.92@tcp Sep 4 18:13:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807592/real 1346807592] req@ffff880037a26000 x1412229540872453/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807617 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:13:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 4 18:13:47 fat-intel-3vm8 kernel: LDISKFS-fs (dm-6): recovery complete Sep 4 18:13:47 fat-intel-3vm8 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:13:47 fat-intel-3vm8 kernel: LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. quota=off. Opts: Sep 4 18:13:47 fat-intel-3vm8 kernel: Lustre: MGC10.10.4.88@tcp: Reactivating import Sep 4 18:14:01 fat-intel-3vm8 ntpd[1784]: synchronized to 10.10.0.1, stratum 2 Sep 4 18:14:07 fat-intel-3vm8 kernel: LustreError: 137-5: UUID 'lustre-OST0006_UUID' is not available for connect (no target) Sep 4 18:14:07 fat-intel-3vm8 kernel: LustreError: Skipped 138 previous similar messages Sep 4 18:14:17 fat-intel-3vm8 kernel: LustreError: 3948:0:(obd_mount.c:1149:server_register_target()) Cannot talk to the MGS: -5, not fatal Sep 4 18:14:23 fat-intel-3vm8 kernel: Lustre: 3975:0:(ldlm_lib.c:2150:target_recovery_init()) RECOVERY: service lustre-OST0006, 4 recoverable clients, last_transno 5950 Sep 4 18:14:23 fat-intel-3vm8 kernel: Lustre: lustre-OST0006: Now serving lustre-OST0006 on /dev/dm-6 with recovery enabled Sep 4 18:14:27 fat-intel-3vm8 kernel: Lustre: lustre-OST0006: Temporarily refusing client connection from 10.10.4.91@tcp Sep 4 18:14:34 fat-intel-3vm8 rshd[3922]: pam_unix(rsh:session): session closed for user root Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3922 duration=91(sec) Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: START: shell pid=3979 from=::ffff:10.10.4.86 Sep 4 18:14:34 fat-intel-3vm8 rshd[3979]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:14:34 fat-intel-3vm8 rshd[3979]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:14:34 fat-intel-3vm8 rshd[3980]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ";echo XXRETCODE:$?' Sep 4 18:14:34 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Sep 4 18:14:34 fat-intel-3vm8 rshd[3979]: pam_unix(rsh:session): session closed for user root Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=3979 duration=0(sec) Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: START: shell pid=4002 from=::ffff:10.10.4.86 Sep 4 18:14:34 fat-intel-3vm8 rshd[4002]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:14:34 fat-intel-3vm8 rshd[4002]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:14:34 fat-intel-3vm8 rshd[4003]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \" 0xffb7e3ff\" 32 ");echo XXRETCODE:$?' Sep 4 18:14:34 fat-intel-3vm8 kernel: LNet: 4108:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Sep 4 18:14:34 fat-intel-3vm8 rshd[4002]: pam_unix(rsh:session): session closed for user root Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=4002 duration=0(sec) Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: START: shell pid=4111 from=::ffff:10.10.4.86 Sep 4 18:14:34 fat-intel-3vm8 rshd[4111]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:14:34 fat-intel-3vm8 rshd[4111]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:14:34 fat-intel-3vm8 rshd[4112]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='/usr/sbin/lctl mark "e2label /dev/lvm-OSS/P7 2>/dev/null";echo XXRETCODE:$?' Sep 4 18:14:34 fat-intel-3vm8 kernel: Lustre: DEBUG MARKER: e2label /dev/lvm-OSS/P7 2>/dev/null Sep 4 18:14:34 fat-intel-3vm8 rshd[4111]: pam_unix(rsh:session): session closed for user root Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=4111 duration=0(sec) Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: START: shell pid=4134 from=::ffff:10.10.4.86 Sep 4 18:14:34 fat-intel-3vm8 rshd[4134]: pam_rhosts(rsh:auth): allowed access to root@fat-intel-3vm1.lab.whamcloud.com as root Sep 4 18:14:34 fat-intel-3vm8 rshd[4134]: pam_unix(rsh:session): session opened for user root by (uid=0) Sep 4 18:14:34 fat-intel-3vm8 rshd[4135]: root@fat-intel-3vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "e2label /dev/lvm-OSS/P7 2>/dev/null");echo XXRETCODE:$?' Sep 4 18:14:34 fat-intel-3vm8 rshd[4134]: pam_unix(rsh:session): session closed for user root Sep 4 18:14:34 fat-intel-3vm8 xinetd[1776]: EXIT: shell status=0 pid=4134 duration=0(sec) Sep 4 18:14:35 fat-intel-3vm8 kernel: Lustre: lustre-OST0006: Will be in recovery for at least 1:00, or until 4 clients reconnect Sep 4 18:14:39 fat-intel-3vm8 kernel: Lustre: lustre-OST0006: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Sep 4 18:14:39 fat-intel-3vm8 kernel: Lustre: lustre-OST0006: received MDS connection from 10.10.4.92@tcp Sep 4 18:18:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807862/real 1346807862] req@ffff880037705800 x1412229540872552/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346807887 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:18:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Sep 4 18:20:01 fat-intel-3vm8 CROND[4191]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 18:27:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346808402/real 1346808402] req@ffff88007c723400 x1412229540872570/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346808427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:27:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Sep 4 18:30:01 fat-intel-3vm8 CROND[4225]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 18:37:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346809032/real 1346809032] req@ffff88007c723c00 x1412229540872591/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346809057 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:37:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Sep 4 18:40:01 fat-intel-3vm8 CROND[4259]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 18:48:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346809662/real 1346809662] req@ffff8800709db800 x1412229540872612/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346809687 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:48:07 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Sep 4 18:50:01 fat-intel-3vm8 CROND[4293]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 18:58:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346810292/real 1346810292] req@ffff880037705c00 x1412229540872633/t0(0) o250->MGC10.10.4.88@tcp@10.10.4.88@tcp:26/25 lens 400/544 e 0 to 1 dl 1346810317 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 4 18:58:37 fat-intel-3vm8 kernel: Lustre: 2075:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Sep 4 19:00:01 fat-intel-3vm8 CROND[4327]: (root) CMD (/usr/lib64/sa/sa1 1 1) Sep 4 19:01:01 fat-intel-3vm8 CROND[4332]: (root) CMD (run-parts /etc/cron.hourly) Sep 4 19:01:01 fat-intel-3vm8 run-parts(/etc/cron.hourly)[4332 starting 0anacron Sep 4 19:01:01 fat-intel-3vm8 run-parts(/etc/cron.hourly)[4341 finished 0anacron Sep 4 19:04:34 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: haven't heard from client 85be9246-99f9-3b19-5f81-ad80152fd420 (at 10.10.4.91@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff88007254fc00, cur 1346810674 expire 1346810644 last 1346810627 Sep 4 19:04:34 fat-intel-3vm8 kernel: Lustre: lustre-OST0004: haven't heard from client f75916a5-eaec-360e-dafc-5ebb1448c443 (at 10.10.4.90@tcp) in 34 seconds. I think it's dead, and I am evicting it. exp ffff88007598b000, cur 1346810674 expire 1346810644 last 1346810640 Sep 4 19:04:39 fat-intel-3vm8 kernel: Lustre: lustre-OST0000: haven't heard from client 85be9246-99f9-3b19-5f81-ad80152fd420 (at 10.10.4.91@tcp) in 52 seconds. I think it's dead, and I am evicting it. exp ffff8800732a6000, cur 1346810679 expire 1346810649 last 1346810627 Sep 4 19:04:39 fat-intel-3vm8 kernel: Lustre: Skipped 10 previous similar messages Sep 4 19:04:40 fat-intel-3vm8 kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4298439482, last ping 4298444482, now 4298449482 Sep 4 19:04:40 fat-intel-3vm8 kernel: connection1:0: detected conn error (1011) Sep 4 19:04:41 fat-intel-3vm8 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3) Sep 4 19:05:38 fat-intel-3vm8 kernel: MTRR variable ranges enabled: Sep 4 19:05:38 fat-intel-3vm8 kernel: 0 base 00E0000000 mask FFE0000000 uncachable Sep 4 19:05:38 fat-intel-3vm8 kernel: 1 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 2 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 3 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 4 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 5 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 6 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: 7 disabled Sep 4 19:05:38 fat-intel-3vm8 kernel: PAT not supported by CPU. Sep 4 19:05:38 fat-intel-3vm8 kernel: initial memory mapped : 0 - 20000000 Sep 4 19:05:38 fat-intel-3vm8 kernel: init_memory_mapping: 0000000000000000-000000007fffd000 Sep 4 19:05:38 fat-intel-3vm8 kernel: 0000000000 - 007fe00000 page 2M Sep 4 19:05:38 fat-intel-3vm8 kernel: 007fe00000 - 007fffd000 page 4k Sep 4 19:05:38 fat-intel-3vm8 kernel: kernel direct mapping tables up to 7fffd000 @ 8000-c000