Lustre: 5939:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448322180/real 1448322180] req@ffff880037ab8680 x1518674503614948/t0(0) o700->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:30/10 lens 264/248 e 0 to 1 dl 1448322187 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 5939:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) == 15:43:36 (1448322216) LustreError: 11-0: lustre-OST0000-osc-ffff8800600ba400: operation obd_ping to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 17 previous similar messages Lustre: lustre-OST0000-osc-ffff8800600ba400: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 40 previous similar messages Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) == 15:44:57 (1448322297) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007d1553c0 x1518674503616100/t176093659260(176093659260) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 816/544 e 0 to 0 dl 1448322328 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 10 previous similar messages Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) == 15:45:22 (1448322322) Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) == 15:45:28 (1448322328) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 21 previous similar messages LustreError: 27468:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight == 15:46:04 (1448322364) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 14 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f57728115e to 0x662ca8f5772813f7 Lustre: Skipped 14 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight == 15:46:32 (1448322392) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight == 15:47:02 (1448322422) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight == 15:47:29 (1448322449) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight == 15:48:00 (1448322480) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight == 15:48:29 (1448322509) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight == 15:49:01 (1448322541) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight == 15:49:28 (1448322568) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 == 15:50:00 (1448322600) Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) == 15:50:06 (1448322606) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op == 15:50:44 (1448322644) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) == 15:51:17 (1448322677) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op == 15:52:11 (1448322731) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op == 15:52:42 (1448322762) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race == 15:53:03 (1448322783) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec Lustre: 27468:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448322804/real 1448322804] req@ffff88007d28f680 x1518674503706020/t0(0) o101->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 328/344 e 0 to 1 dl 1448322811 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 27468:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 74 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink == 15:54:11 (1448322851) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 34 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup == 15:54:42 (1448322882) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 LustreError: 11-0: lustre-OST0000-osc-ffff8800600ba400: operation obd_ping to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 8 previous similar messages Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup == 15:55:59 (1448322959) Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 20 previous similar messages LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 14 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f5772d74ce to 0x662ca8f5772d769c Lustre: Skipped 14 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup == 15:56:51 (1448323011) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay == 15:57:19 (1448323039) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2844:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff88007adf03c0 x1518674503731144/t253403070475(253403070475) o36->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 640/416 e 0 to 1 dl 1448323073 ref 2 fl Interpret:EX/4/ffffffff rc -110/-1 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007adf00c0 x1518674503731168/t253403070478(253403070478) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 816/544 e 0 to 0 dl 1448323080 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 19 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec Lustre: DEBUG MARKER: SKIP: replay-single test_65a skipping SLOW test 65a (base 65) Lustre: DEBUG MARKER: SKIP: replay-single test_65b skipping SLOW test 65b (base 65) Lustre: DEBUG MARKER: SKIP: replay-single test_66a skipping SLOW test 66a (base 66) Lustre: DEBUG MARKER: SKIP: replay-single test_66b skipping SLOW test 66b (base 66) Lustre: DEBUG MARKER: SKIP: replay-single test_67a skipping SLOW test 67a (base 67) Lustre: DEBUG MARKER: SKIP: replay-single test_67b skipping SLOW test 67b (base 67) Lustre: DEBUG MARKER: SKIP: replay-single test_68 skipping SLOW test 68 Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f == 15:57:57 (1448323077) Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; clients == 15:58:00 (1448323080) Lustre: DEBUG MARKER: Started rundbench load pid=32590 ... Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec LustreError: 27468:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery == 16:00:28 (1448323228) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70c fail mds1 1 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70c fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800600ba400: operation ldlm_enqueue to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 10 previous similar messages Lustre: 19960:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448323527/real 1448323527] req@ffff88004c5f9080 x1518674504632444/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448323534 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 19960:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 22 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery == 16:06:05 (1448323565) Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails == 16:06:06 (1448323566) Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery == 16:06:07 (1448323567) Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close == 16:06:08 (1448323568) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 19956:0:(client.c:2844:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff88007b2850c0 x1518674503732356/t257698037803(257698037803) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 872/544 e 0 to 1 dl 1448323604 ref 2 fl Interpret:EXP/4/ffffffff rc -110/-1 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close == 16:06:47 (1448323607) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery == 16:07:11 (1448323631) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 == 16:07:47 (1448323667) Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 == 16:07:48 (1448323668) Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] == 16:07:49 (1448323669) Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs == 16:07:50 (1448323670) Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 == 16:07:51 (1448323671) Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 == 16:07:52 (1448323672) Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 == 16:07:53 (1448323673) Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs == 16:07:54 (1448323674) Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 == 16:07:55 (1448323675) Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 == 16:07:56 (1448323676) Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 == 16:07:57 (1448323677) Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs == 16:07:58 (1448323678) Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 == 16:07:59 (1448323679) Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 == 16:08:00 (1448323680) Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 == 16:08:01 (1448323681) Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs == 16:08:02 (1448323682) Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect == 16:08:03 (1448323683) LustreError: 167-0: lustre-MDT0000-mdc-ffff88007b864c00: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message LustreError: 22544:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007b864c00), error -5 LustreError: 22544:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) == 16:08:06 (1448323686) Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f577544984 to 0x662ca8f577547591 Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) == 16:08:36 (1448323716) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG == 16:09:29 (1448323769) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 87: write replay == 16:09:40 (1448323780) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) == 16:10:02 (1448323802) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files == 16:10:26 (1448323826) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007c3303c0 x1518674504704844/t309237645361(309237645361) o101->lustre-MDT0000-mdc-ffff880049ff8800@10.100.4.183@tcp:12/10 lens 800/544 e 0 to 0 dl 1448323863 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 1486 previous similar messages Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection == 16:11:05 (1448323865) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments == 16:12:57 (1448323977) Lustre: DEBUG MARKER: == replay-single test 93: replay + reconnect == 16:13:14 (1448323994) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 43 sec Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 == 16:14:12 (1448324052) Lustre: DEBUG MARKER: SKIP: replay-single test_100a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 == 16:14:13 (1448324053) Lustre: DEBUG MARKER: SKIP: replay-single test_100b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery == 16:14:14 (1448324054) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 167-0: lustre-MDT0000-mdc-ffff88007b6d9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: 1724:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007b6d9800), error -5 LustreError: 1724:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight == 16:14:55 (1448324095) Lustre: 3158:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324095/real 1448324095] req@ffff88007d00a380 x1518674504790516/t0(0) o36->lustre-MDT0000-mdc-ffff88007b6d9800@10.100.4.183@tcp:12/10 lens 608/1016 e 0 to 1 dl 1448324139 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3158:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight == 16:15:40 (1448324140) Lustre: lustre-MDT0000-mdc-ffff88007b6d9800: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 20 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight == 16:16:25 (1448324185) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007b6d9800: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 27 previous similar messages Lustre: lustre-MDT0000-mdc-ffff88007b6d9800: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Skipped 15 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102d: check replay LustreError: 30067:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 == 16:17:21 (1448324241) Lustre: DEBUG MARKER: SKIP: replay-single test_110a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client == 16:17:22 (1448324242) Lustre: DEBUG MARKER: SKIP: replay-single test_110b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 == 16:17:23 (1448324243) Lustre: DEBUG MARKER: SKIP: replay-single test_110c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client == 16:17:24 (1448324244) Lustre: DEBUG MARKER: SKIP: replay-single test_110d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 == 16:17:25 (1448324245) Lustre: DEBUG MARKER: SKIP: replay-single test_110e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110f: DNE: create striped dir, fail MDT1/MDT2 == 16:17:26 (1448324246) Lustre: DEBUG MARKER: SKIP: replay-single test_110f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 == 16:17:27 (1448324247) Lustre: DEBUG MARKER: SKIP: replay-single test_110g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 == 16:17:28 (1448324248) Lustre: DEBUG MARKER: SKIP: replay-single test_111a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 == 16:17:29 (1448324249) Lustre: DEBUG MARKER: SKIP: replay-single test_111b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 == 16:17:30 (1448324250) Lustre: DEBUG MARKER: SKIP: replay-single test_111c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 == 16:17:31 (1448324251) Lustre: DEBUG MARKER: SKIP: replay-single test_111d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 == 16:17:32 (1448324252) Lustre: DEBUG MARKER: SKIP: replay-single test_111e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 == 16:17:33 (1448324253) Lustre: DEBUG MARKER: SKIP: replay-single test_111f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 == 16:17:34 (1448324254) Lustre: DEBUG MARKER: SKIP: replay-single test_111g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 == 16:17:35 (1448324255) Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 == 16:17:36 (1448324256) Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 == 16:17:37 (1448324257) Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 == 16:17:38 (1448324258) Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 == 16:17:39 (1448324259) Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 == 16:17:40 (1448324260) Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 == 16:17:41 (1448324261) Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 == 16:17:41 (1448324261) Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 == 16:17:42 (1448324262) Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 == 16:17:43 (1448324263) Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 == 16:17:44 (1448324264) Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 == 16:17:45 (1448324265) Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 == 16:17:46 (1448324266) Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 == 16:17:47 (1448324267) Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory == 16:17:48 (1448324268) Lustre: DEBUG MARKER: SKIP: replay-single test_115 needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery == 16:17:49 (1448324269) Lustre: DEBUG MARKER: SKIP: replay-single test_116a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery == 16:17:50 (1448324270) Lustre: DEBUG MARKER: SKIP: replay-single test_116b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test complete, duration 3374 sec == 16:17:51 (1448324271) LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: 22117:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp SELinux: initialized (dev 0:19, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1a, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1b, type nfs4), uses genfs_contexts LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 22602:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 22602:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Nov 23 16:24:31 PST 2015 Lustre: DEBUG MARKER: excepting tests: 32newtarball Lustre: DEBUG MARKER: skipping tests SLOW=no: 30a 31 45 69 Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup == 16:25:20 (1448324720) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: 27240:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) == 16:25:52 (1448324752) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) == 16:26:35 (1448324795) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) == 16:27:15 (1448324835) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup == 16:28:00 (1448324880) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup == 16:28:46 (1448324926) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr LustreError: 11-0: lustre-MDT0000-mdc-ffff880061b91c00: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff880061b91c00: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: 31701:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324953/real 1448324953] req@ffff880079eb3980 x1518678769139852/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448324960 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 31783:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324957/real 1448324957] req@ffff880079eb3680 x1518678769139880/t0(0) o251->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448324963 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) == 16:29:39 (1448324979) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 32542:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448325040/real 1448325040] req@ffff88007a0d1c80 x1518678877143044/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448325045 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 32542:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448325065/real 1448325065] req@ffff88007a0d1c80 x1518678877143060/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448325075 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 15c-8: MGC10.100.4.183@tcp: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Lustre: Unmounted lustre-client LustreError: 32525:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-5) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) == 16:31:26 (1448325086) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 LustreError: 950:0:(mgc_request.c:329:config_log_add()) logname wrong.lustre-client is too long LustreError: 15b-f: MGC10.100.4.183@tcp: The configuration from log 'wrong.lustre-client'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. Lustre: Unmounted wrong.lustre-client LustreError: 950:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-22) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down == 16:31:51 (1448325111) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: 1996:0:(lmv_obd.c:299:lmv_init_ea_size()) lustre-clilmv-ffff880037daec00: NULL export for 0 Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) == 16:33:21 (1448325201) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 LustreError: 2843:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 sleeping for 10000ms LustreError: 2843:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 awake Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) == 16:34:05 (1448325245) Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again == 16:34:14 (1448325254) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup == 16:34:46 (1448325286) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup == 16:35:28 (1448325328) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs == 16:36:09 (1448325369) Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) == 16:37:12 (1448325432) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 8443:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 8443:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals == 16:39:19 (1448325559) Lustre: DEBUG MARKER: use device /dev/vda5 with MIN=2000000 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M Lustre: Unmounted lustre-client Lustre: 11466:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 11833:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 11833:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs == 16:41:02 (1448325662) Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS == 16:41:11 (1448325671) Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab == 16:42:06 (1448325726) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Remounted lustre-client read-only Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first == 16:42:40 (1448325760) Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first == 16:43:05 (1448325785) Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last == 16:44:08 (1448325848) Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds == 16:45:35 (1448325935) Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device Lustre: DEBUG MARKER: == conf-sanity test 21e: separate MGS and MDS == 16:45:36 (1448325936) Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) == 16:45:37 (1448325937) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: 18856:0:(lmv_obd.c:299:lmv_init_ea_size()) lustre-clilmv-ffff88006ebea000: NULL export for 0 Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800619c6400.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: 19484:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay == 16:46:20 (1448325980) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: 20160:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326002/real 1448326002] req@ffff88004c665080 x1518679875387488/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326009 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 20232:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326005/real 1448326005] req@ffff88006e72c680 x1518679875387504/t0(0) o251->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326011 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d347c00: operation mds_connect to node 10.100.4.183@tcp failed: rc = -16 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d347c00: operation mds_connect to node 10.100.4.183@tcp failed: rc = -16 LustreError: 20367:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007d347c00), error -16 LustreError: 20145:0:(lov_obd.c:920:lov_cleanup()) lustre-clilov-ffff88007d347c00: lov tgt 0 not cleaned! deathrow=0, lovrc=1 Lustre: Unmounted lustre-client LustreError: 20367:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-16) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount == 16:47:35 (1448326055) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node == 16:48:14 (1448326094) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24a mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) == 16:48:23 (1448326103) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24b mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced == 16:48:24 (1448326104) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) == 16:48:56 (1448326136) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 23920:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 23920:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: 24331:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first == 16:49:29 (1448326169) LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover == 16:51:00 (1448326260) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: 25875:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326281/real 1448326281] req@ffff88007b2a0380 x1518680167940192/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326288 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: 25875:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326286/real 1448326286] req@ffff88007b2a0080 x1518680167940196/t0(0) o400->lustre-MDT0000-mdc-ffff880061afe000@10.100.4.183@tcp:12/10 lens 224/224 e 0 to 1 dl 1448326293 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: lustre-MDT0000-mdc-ffff880061afe000: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 25872:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326288/real 1448326288] req@ffff88007b04cc80 x1518680167940208/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448326294 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f57758fb70 to 0x662ca8f57758fd0d Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: 25872:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326293/real 1448326293] req@ffff88007b04c380 x1518680167940220/t0(0) o38->lustre-MDT0000-mdc-ffff880061afe000@10.100.4.183@tcp:12/10 lens 520/544 e 0 to 1 dl 1448326299 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: lustre-MDT0000-mdc-ffff880061afe000: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting == 16:52:06 (1448326326) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 28a: set symlink parameters permanently with conf_param == 16:53:08 (1448326388) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST == 16:54:41 (1448326481) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: SKIP: conf-sanity test_30a skipping SLOW test 30a Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids == 16:56:09 (1448326569) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: SKIP: conf-sanity test_31 skipping SLOW test 31 Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 16:56:55 (1448326615) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 32311:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 32311:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade with writeconf == 17:06:54 (1448327214) Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: test_32b failed with 4