Lustre: 5939:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448322180/real 1448322180] req@ffff880037ab8680 x1518674503614948/t0(0) o700->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:30/10 lens 264/248 e 0 to 1 dl 1448322187 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 5939:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) == 15:43:36 (1448322216) LustreError: 11-0: lustre-OST0000-osc-ffff8800600ba400: operation obd_ping to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 17 previous similar messages Lustre: lustre-OST0000-osc-ffff8800600ba400: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 40 previous similar messages Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) == 15:44:57 (1448322297) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007d1553c0 x1518674503616100/t176093659260(176093659260) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 816/544 e 0 to 0 dl 1448322328 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 10 previous similar messages Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) == 15:45:22 (1448322322) Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) == 15:45:28 (1448322328) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 21 previous similar messages LustreError: 27468:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight == 15:46:04 (1448322364) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 14 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f57728115e to 0x662ca8f5772813f7 Lustre: Skipped 14 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight == 15:46:32 (1448322392) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight == 15:47:02 (1448322422) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight == 15:47:29 (1448322449) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight == 15:48:00 (1448322480) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight == 15:48:29 (1448322509) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight == 15:49:01 (1448322541) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight == 15:49:28 (1448322568) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 == 15:50:00 (1448322600) Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) == 15:50:06 (1448322606) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op == 15:50:44 (1448322644) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) == 15:51:17 (1448322677) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op == 15:52:11 (1448322731) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op == 15:52:42 (1448322762) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race == 15:53:03 (1448322783) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec Lustre: 27468:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448322804/real 1448322804] req@ffff88007d28f680 x1518674503706020/t0(0) o101->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 328/344 e 0 to 1 dl 1448322811 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 27468:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 74 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink == 15:54:11 (1448322851) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 34 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup == 15:54:42 (1448322882) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 LustreError: 11-0: lustre-OST0000-osc-ffff8800600ba400: operation obd_ping to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 8 previous similar messages Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup == 15:55:59 (1448322959) Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 20 previous similar messages LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 14 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f5772d74ce to 0x662ca8f5772d769c Lustre: Skipped 14 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup == 15:56:51 (1448323011) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay == 15:57:19 (1448323039) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2844:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff88007adf03c0 x1518674503731144/t253403070475(253403070475) o36->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 640/416 e 0 to 1 dl 1448323073 ref 2 fl Interpret:EX/4/ffffffff rc -110/-1 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007adf00c0 x1518674503731168/t253403070478(253403070478) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 816/544 e 0 to 0 dl 1448323080 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 19 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec Lustre: DEBUG MARKER: SKIP: replay-single test_65a skipping SLOW test 65a (base 65) Lustre: DEBUG MARKER: SKIP: replay-single test_65b skipping SLOW test 65b (base 65) Lustre: DEBUG MARKER: SKIP: replay-single test_66a skipping SLOW test 66a (base 66) Lustre: DEBUG MARKER: SKIP: replay-single test_66b skipping SLOW test 66b (base 66) Lustre: DEBUG MARKER: SKIP: replay-single test_67a skipping SLOW test 67a (base 67) Lustre: DEBUG MARKER: SKIP: replay-single test_67b skipping SLOW test 67b (base 67) Lustre: DEBUG MARKER: SKIP: replay-single test_68 skipping SLOW test 68 Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f == 15:57:57 (1448323077) Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; clients == 15:58:00 (1448323080) Lustre: DEBUG MARKER: Started rundbench load pid=32590 ... Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec LustreError: 27468:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery == 16:00:28 (1448323228) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70c fail mds1 1 times Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70c fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800600ba400: operation ldlm_enqueue to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 10 previous similar messages Lustre: 19960:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448323527/real 1448323527] req@ffff88004c5f9080 x1518674504632444/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448323534 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 19960:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 22 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery == 16:06:05 (1448323565) Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails == 16:06:06 (1448323566) Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery == 16:06:07 (1448323567) Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close == 16:06:08 (1448323568) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800600ba400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 19956:0:(client.c:2844:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff88007b2850c0 x1518674503732356/t257698037803(257698037803) o101->lustre-MDT0000-mdc-ffff8800600ba400@10.100.4.183@tcp:12/10 lens 872/544 e 0 to 1 dl 1448323604 ref 2 fl Interpret:EXP/4/ffffffff rc -110/-1 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close == 16:06:47 (1448323607) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery == 16:07:11 (1448323631) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 == 16:07:47 (1448323667) Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 == 16:07:48 (1448323668) Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] == 16:07:49 (1448323669) Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs == 16:07:50 (1448323670) Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 == 16:07:51 (1448323671) Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 == 16:07:52 (1448323672) Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 == 16:07:53 (1448323673) Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs == 16:07:54 (1448323674) Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 == 16:07:55 (1448323675) Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 == 16:07:56 (1448323676) Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 == 16:07:57 (1448323677) Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs == 16:07:58 (1448323678) Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 == 16:07:59 (1448323679) Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 == 16:08:00 (1448323680) Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 == 16:08:01 (1448323681) Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs == 16:08:02 (1448323682) Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect == 16:08:03 (1448323683) LustreError: 167-0: lustre-MDT0000-mdc-ffff88007b864c00: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message LustreError: 22544:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007b864c00), error -5 LustreError: 22544:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) == 16:08:06 (1448323686) Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f577544984 to 0x662ca8f577547591 Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) == 16:08:36 (1448323716) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG == 16:09:29 (1448323769) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 87: write replay == 16:09:40 (1448323780) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) == 16:10:02 (1448323802) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files == 16:10:26 (1448323826) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007c3303c0 x1518674504704844/t309237645361(309237645361) o101->lustre-MDT0000-mdc-ffff880049ff8800@10.100.4.183@tcp:12/10 lens 800/544 e 0 to 0 dl 1448323863 ref 2 fl Interpret:R/4/0 rc 301/301 LustreError: 19956:0:(client.c:2899:ptlrpc_replay_interpret()) Skipped 1486 previous similar messages Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection == 16:11:05 (1448323865) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments == 16:12:57 (1448323977) Lustre: DEBUG MARKER: == replay-single test 93: replay + reconnect == 16:13:14 (1448323994) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 43 sec Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 == 16:14:12 (1448324052) Lustre: DEBUG MARKER: SKIP: replay-single test_100a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 == 16:14:13 (1448324053) Lustre: DEBUG MARKER: SKIP: replay-single test_100b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery == 16:14:14 (1448324054) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 167-0: lustre-MDT0000-mdc-ffff88007b6d9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: 1724:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007b6d9800), error -5 LustreError: 1724:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight == 16:14:55 (1448324095) Lustre: 3158:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324095/real 1448324095] req@ffff88007d00a380 x1518674504790516/t0(0) o36->lustre-MDT0000-mdc-ffff88007b6d9800@10.100.4.183@tcp:12/10 lens 608/1016 e 0 to 1 dl 1448324139 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3158:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight == 16:15:40 (1448324140) Lustre: lustre-MDT0000-mdc-ffff88007b6d9800: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 20 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight == 16:16:25 (1448324185) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007b6d9800: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 27 previous similar messages Lustre: lustre-MDT0000-mdc-ffff88007b6d9800: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Skipped 15 previous similar messages Lustre: DEBUG MARKER: == replay-single test 102d: check replay LustreError: 30067:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 == 16:17:21 (1448324241) Lustre: DEBUG MARKER: SKIP: replay-single test_110a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client == 16:17:22 (1448324242) Lustre: DEBUG MARKER: SKIP: replay-single test_110b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 == 16:17:23 (1448324243) Lustre: DEBUG MARKER: SKIP: replay-single test_110c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client == 16:17:24 (1448324244) Lustre: DEBUG MARKER: SKIP: replay-single test_110d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 == 16:17:25 (1448324245) Lustre: DEBUG MARKER: SKIP: replay-single test_110e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110f: DNE: create striped dir, fail MDT1/MDT2 == 16:17:26 (1448324246) Lustre: DEBUG MARKER: SKIP: replay-single test_110f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 == 16:17:27 (1448324247) Lustre: DEBUG MARKER: SKIP: replay-single test_110g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 == 16:17:28 (1448324248) Lustre: DEBUG MARKER: SKIP: replay-single test_111a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 == 16:17:29 (1448324249) Lustre: DEBUG MARKER: SKIP: replay-single test_111b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 == 16:17:30 (1448324250) Lustre: DEBUG MARKER: SKIP: replay-single test_111c needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 == 16:17:31 (1448324251) Lustre: DEBUG MARKER: SKIP: replay-single test_111d needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 == 16:17:32 (1448324252) Lustre: DEBUG MARKER: SKIP: replay-single test_111e needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 == 16:17:33 (1448324253) Lustre: DEBUG MARKER: SKIP: replay-single test_111f needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 == 16:17:34 (1448324254) Lustre: DEBUG MARKER: SKIP: replay-single test_111g needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 == 16:17:35 (1448324255) Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 == 16:17:36 (1448324256) Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 == 16:17:37 (1448324257) Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 == 16:17:38 (1448324258) Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 == 16:17:39 (1448324259) Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 == 16:17:40 (1448324260) Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 == 16:17:41 (1448324261) Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 == 16:17:41 (1448324261) Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 == 16:17:42 (1448324262) Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 == 16:17:43 (1448324263) Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 == 16:17:44 (1448324264) Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 == 16:17:45 (1448324265) Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 == 16:17:46 (1448324266) Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 == 16:17:47 (1448324267) Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory == 16:17:48 (1448324268) Lustre: DEBUG MARKER: SKIP: replay-single test_115 needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery == 16:17:49 (1448324269) Lustre: DEBUG MARKER: SKIP: replay-single test_116a needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery == 16:17:50 (1448324270) Lustre: DEBUG MARKER: SKIP: replay-single test_116b needs >= 2 MDTs Lustre: DEBUG MARKER: == replay-single test complete, duration 3374 sec == 16:17:51 (1448324271) LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: 22117:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp SELinux: initialized (dev 0:19, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1a, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1b, type nfs4), uses genfs_contexts LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 22602:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 22602:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Nov 23 16:24:31 PST 2015 Lustre: DEBUG MARKER: excepting tests: 32newtarball Lustre: DEBUG MARKER: skipping tests SLOW=no: 30a 31 45 69 Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup == 16:25:20 (1448324720) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: 27240:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) == 16:25:52 (1448324752) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) == 16:26:35 (1448324795) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) == 16:27:15 (1448324835) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup == 16:28:00 (1448324880) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup == 16:28:46 (1448324926) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr LustreError: 11-0: lustre-MDT0000-mdc-ffff880061b91c00: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff880061b91c00: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: 31701:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324953/real 1448324953] req@ffff880079eb3980 x1518678769139852/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448324960 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 31783:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448324957/real 1448324957] req@ffff880079eb3680 x1518678769139880/t0(0) o251->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448324963 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) == 16:29:39 (1448324979) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 32542:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448325040/real 1448325040] req@ffff88007a0d1c80 x1518678877143044/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448325045 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 32542:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448325065/real 1448325065] req@ffff88007a0d1c80 x1518678877143060/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448325075 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 15c-8: MGC10.100.4.183@tcp: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Lustre: Unmounted lustre-client LustreError: 32525:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-5) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) == 16:31:26 (1448325086) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 LustreError: 950:0:(mgc_request.c:329:config_log_add()) logname wrong.lustre-client is too long LustreError: 15b-f: MGC10.100.4.183@tcp: The configuration from log 'wrong.lustre-client'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. Lustre: Unmounted wrong.lustre-client LustreError: 950:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-22) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down == 16:31:51 (1448325111) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: 1996:0:(lmv_obd.c:299:lmv_init_ea_size()) lustre-clilmv-ffff880037daec00: NULL export for 0 Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) == 16:33:21 (1448325201) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 LustreError: 2843:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 sleeping for 10000ms LustreError: 2843:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 awake Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) == 16:34:05 (1448325245) Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again == 16:34:14 (1448325254) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup == 16:34:46 (1448325286) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup == 16:35:28 (1448325328) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs == 16:36:09 (1448325369) Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) == 16:37:12 (1448325432) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 8443:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 8443:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals == 16:39:19 (1448325559) Lustre: DEBUG MARKER: use device /dev/vda5 with MIN=2000000 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M Lustre: Unmounted lustre-client Lustre: 11466:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 11833:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 11833:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs == 16:41:02 (1448325662) Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS == 16:41:11 (1448325671) Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab == 16:42:06 (1448325726) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Remounted lustre-client read-only Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first == 16:42:40 (1448325760) Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first == 16:43:05 (1448325785) Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last == 16:44:08 (1448325848) Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds == 16:45:35 (1448325935) Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device Lustre: DEBUG MARKER: == conf-sanity test 21e: separate MGS and MDS == 16:45:36 (1448325936) Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) == 16:45:37 (1448325937) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: 18856:0:(lmv_obd.c:299:lmv_init_ea_size()) lustre-clilmv-ffff88006ebea000: NULL export for 0 Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800619c6400.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: 19484:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay == 16:46:20 (1448325980) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: 20160:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326002/real 1448326002] req@ffff88004c665080 x1518679875387488/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326009 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 20232:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326005/real 1448326005] req@ffff88006e72c680 x1518679875387504/t0(0) o251->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326011 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d347c00: operation mds_connect to node 10.100.4.183@tcp failed: rc = -16 LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d347c00: operation mds_connect to node 10.100.4.183@tcp failed: rc = -16 LustreError: 20367:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007d347c00), error -16 LustreError: 20145:0:(lov_obd.c:920:lov_cleanup()) lustre-clilov-ffff88007d347c00: lov tgt 0 not cleaned! deathrow=0, lovrc=1 Lustre: Unmounted lustre-client LustreError: 20367:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-16) LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount == 16:47:35 (1448326055) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node == 16:48:14 (1448326094) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24a mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) == 16:48:23 (1448326103) Lustre: DEBUG MARKER: SKIP: conf-sanity test_24b mixed loopback and real device not working Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced == 16:48:24 (1448326104) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) == 16:48:56 (1448326136) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 23920:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 23920:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: 24331:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first == 16:49:29 (1448326169) LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover == 16:51:00 (1448326260) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: 25875:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326281/real 1448326281] req@ffff88007b2a0380 x1518680167940192/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448326288 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: 25875:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326286/real 1448326286] req@ffff88007b2a0080 x1518680167940196/t0(0) o400->lustre-MDT0000-mdc-ffff880061afe000@10.100.4.183@tcp:12/10 lens 224/224 e 0 to 1 dl 1448326293 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: lustre-MDT0000-mdc-ffff880061afe000: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 25872:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326288/real 1448326288] req@ffff88007b04cc80 x1518680167940208/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1448326294 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x662ca8f57758fb70 to 0x662ca8f57758fd0d Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: 25872:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448326293/real 1448326293] req@ffff88007b04c380 x1518680167940220/t0(0) o38->lustre-MDT0000-mdc-ffff880061afe000@10.100.4.183@tcp:12/10 lens 520/544 e 0 to 1 dl 1448326299 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: lustre-MDT0000-mdc-ffff880061afe000: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting == 16:52:06 (1448326326) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 28a: set symlink parameters permanently with conf_param == 16:53:08 (1448326388) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST == 16:54:41 (1448326481) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: SKIP: conf-sanity test_30a skipping SLOW test 30a Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids == 16:56:09 (1448326569) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: DEBUG MARKER: SKIP: conf-sanity test_31 skipping SLOW test 31 Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 16:56:55 (1448326615) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 32311:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 32311:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade with writeconf == 17:06:54 (1448327214) Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: Mounted t32fs-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: list verification failed Lustre: Unmounted t32fs-client Lustre: DEBUG MARKER: conf-sanity test_32b: @@@@@@ FAIL: test_32b failed with 4 Lustre: 6688:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp SELinux: initialized (dev 0:19, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1a, type nfs4), uses genfs_contexts SELinux: initialized (dev 0:1b, type nfs4), uses genfs_contexts LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-3240-g049252c-PRISTINE-2.6.32-431.29.2.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 13039:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 13039:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Nov 24 08:12:39 PST 2015 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 08:12:44 (1448381564) Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 08:12:45 (1448381565) Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 08:12:46 (1448381566) Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 08:12:46 (1448381566) Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ====================== 08:12:47 (1448381567) Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ======================= 08:12:48 (1448381568) Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) == 08:12:49 (1448381569) Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============== 08:12:49 (1448381569) Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ==== 08:12:50 (1448381570) Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ==== 08:12:51 (1448381571) Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ==== 08:12:52 (1448381572) Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? == 08:12:53 (1448381573) Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) == 08:12:53 (1448381573) Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ====== 08:12:54 (1448381574) Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ======= 08:12:55 (1448381575) Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ========= 08:12:55 (1448381575) Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================== 08:12:56 (1448381576) Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ======== 08:12:57 (1448381577) Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============== 08:12:58 (1448381578) Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ========== 08:12:58 (1448381578) Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ========== 08:12:59 (1448381579) Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ========= 08:13:00 (1448381580) Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ============ 08:13:01 (1448381581) Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f =================== 08:13:01 (1448381581) Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ==================== 08:13:02 (1448381582) Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ================ 08:13:03 (1448381583) Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ====== 08:13:04 (1448381584) Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ========================== 08:13:04 (1448381584) Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ====== 08:13:05 (1448381585) Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================== 08:13:06 (1448381586) Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries == 08:13:07 (1448381587) Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug == 08:13:08 (1448381588) Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink == 08:13:09 (1448381589) LustreError: 29092:0:(symlink.c:98:ll_readlink_internal()) lustre: inode [0x200000401:0x4e:0x0]: symlink length 33 not expected 35 Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled =========================== 08:13:10 (1448381590) Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========== 08:13:11 (1448381591) Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink == 08:13:11 (1448381591) LustreError: 11-0: lustre-MDT0000-mdc-ffff88007da03400: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff88007da03400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 13028:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448381613/real 1448381613] req@ffff88004c72e3c0 x1518738110190600/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448381620 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x18d6cd0c95bc1de0 to 0x18d6cd0c95be1d57 Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) LustreError: 13026:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880063b11c80 x1518738110155392/t4294967548(4294967548) o101->lustre-MDT0000-mdc-ffff88007da03400@10.100.4.183@tcp:12/10 lens 872/544 e 0 to 0 dl 1448381632 ref 2 fl Interpret:RP/4/0 rc 301/301 Lustre: lustre-MDT0000-mdc-ffff88007da03400: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir == 08:13:45 (1448381625) Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 08:13:47 (1448381627) Lustre: 13027:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448381635/real 1448381635] req@ffff88007b4c99c0 x1518738110190792/t0(0) o400->lustre-MDT0000-mdc-ffff88007da03400@10.100.4.183@tcp:12/10 lens 224/224 e 0 to 1 dl 1448381642 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: lustre-MDT0000-mdc-ffff88007da03400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: 13028:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448381640/real 1448381640] req@ffff88007b1a1080 x1518738110190800/t0(0) o400->lustre-MDT0000-mdc-ffff88007da03400@10.100.4.183@tcp:12/10 lens 224/224 e 0 to 1 dl 1448381647 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13028:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x18d6cd0c95be1d57 to 0x18d6cd0c95be2052 Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ================================ 08:14:14 (1448381654) Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ============= 08:14:15 (1448381655) Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ================ 08:14:16 (1448381656) Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ==== 08:14:16 (1448381656) Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================ 08:14:17 (1448381657) Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... ============================= 08:14:18 (1448381658) Lustre: DEBUG MARKER: 1 done Lustre: DEBUG MARKER: 2 done Lustre: DEBUG MARKER: 3 done Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================== 08:14:19 (1448381659) Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user ================= 08:14:20 (1448381660) Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============================ 08:14:21 (1448381661) Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ============================ 08:14:22 (1448381662) Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target == 08:14:22 (1448381662) Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target == 08:14:23 (1448381663) Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target == 08:14:24 (1448381664) Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 08:14:25 (1448381665) Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ======== 08:14:25 (1448381665) Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ====== 08:14:26 (1448381666) Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ======== 08:14:27 (1448381667) Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e == 08:14:28 (1448381668) Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a == 08:14:28 (1448381668) Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================== 08:14:29 (1448381669) Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ========= 08:14:30 (1448381670) Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ========================== 08:14:31 (1448381671) Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =========== 08:14:31 (1448381671) Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) == 08:14:32 (1448381672) Lustre: DEBUG MARKER: SKIP: sanity test_24o skipping SLOW test 24o Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b == 08:14:33 (1448381673) Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ===== 08:14:34 (1448381674) Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ======= 08:14:34 (1448381674) Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c === 08:14:35 (1448381675) Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a === 08:14:36 (1448381676) Lustre: DEBUG MARKER: == sanity test 24u: create stripe file == 08:14:37 (1448381677) Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) == 08:14:38 (1448381678) Lustre: DEBUG MARKER: SKIP: sanity test_24v not enough free inodes 98608 required 100000 Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb == 08:14:38 (1448381678) Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link == 08:14:39 (1448381679) Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed == 08:14:40 (1448381680) Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. == 08:14:41 (1448381681) Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries == 08:15:22 (1448381722) Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir == 08:15:23 (1448381723) Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link == 08:15:24 (1448381724) Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ================= 08:15:25 (1448381725) Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ================= 08:15:26 (1448381726) Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ========================= 08:15:27 (1448381727) Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ======== 08:15:27 (1448381727) Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================== 08:15:28 (1448381728) Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ======== 08:15:29 (1448381729) Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ======== 08:15:30 (1448381730) Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink === 08:15:30 (1448381730) Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ==================================== 08:15:31 (1448381731) Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 08:15:32 (1448381732) Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================== 08:15:33 (1448381733) Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ======== 08:15:34 (1448381734) Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) == 08:15:35 (1448381735) Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects == 08:15:35 (1448381735) Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects == 08:15:36 (1448381736) Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) == 08:15:37 (1448381737) Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps =============== 08:15:38 (1448381738) Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) == 08:15:39 (1448381739) Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ==================== 08:15:40 (1448381740) Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ====== 08:16:13 (1448381773) Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ===== 08:16:59 (1448381819) Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ===== 08:17:33 (1448381853) Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) === 08:18:19 (1448381899) Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) == 08:18:53 (1448381933) Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly == 08:18:53 (1448381933) Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ============ 08:18:54 (1448381934) Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST =================== 08:19:14 (1448381954) Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option == 08:20:10 (1448382010) Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options == 08:20:11 (1448382011) Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 08:20:11 (1448382011) Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive == 08:20:23 (1448382023) Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems == 08:20:45 (1448382045) Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1fc2 ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1fc2, count: 1 Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:314/0x13a seq:0 Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1fc4 ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1fc4, count: 2 Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1251/0x4e3 seq:0 LustreError: 11-0: lustre-OST0001-osc-ffff88007da03400: operation obd_ping to node 10.100.4.184@tcp failed: rc = -107 Lustre: lustre-OST0001-osc-ffff88007da03400: Connection to lustre-OST0001 (at 10.100.4.184@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000-osc-ffff88007da03400: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:315/0x13b seq:0 Lustre: lustre-OST0000-osc-ffff88007da03400: Connection to lustre-OST0000 (at 10.100.4.184@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0001-osc-ffff88007da03400: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values == 08:21:02 (1448382062) Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 08:21:03 (1448382063) Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs == 08:21:03 (1448382063) Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API == 08:21:04 (1448382064) Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases == 08:21:43 (1448382103) Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============== 08:21:44 (1448382104) Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================== 08:21:45 (1448382105) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: first d29 Lustre: DEBUG MARKER: second d29 Lustre: DEBUG MARKER: done Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ================ 08:21:46 (1448382106) Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ============= 08:21:47 (1448382107) Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ====== 08:21:48 (1448382108) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ==================================== 08:21:49 (1448382109) Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ========= 08:21:50 (1448382110) Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links =============== 08:21:50 (1448382110) Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =========================== 08:21:51 (1448382111) Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ================= 08:21:52 (1448382112) Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ===== 08:21:53 (1448382113) Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================= 08:22:00 (1448382120) Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================= 08:22:01 (1448382121) Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================= 08:22:01 (1448382121) Lustre: DEBUG MARKER: == sanity test 31j: link for directory================= 08:22:02 (1448382122) Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir================= 08:22:03 (1448382123) Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir================= 08:22:04 (1448382124) Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file == 08:22:04 (1448382124) Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename == 08:22:05 (1448382125) Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory == 08:22:21 (1448382141) Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ======================= 08:22:22 (1448382142) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ======================= 08:22:23 (1448382143) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir =========== 08:22:24 (1448382144) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir =========== 08:22:25 (1448382145) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ====== 08:22:26 (1448382146) Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ====== 08:22:27 (1448382147) Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 == 08:22:27 (1448382147) Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 == 08:22:28 (1448382148) Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file ============= 08:22:29 (1448382149) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file ============= 08:22:30 (1448382150) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ========== 08:22:31 (1448382151) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ========== 08:22:32 (1448382152) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ======== 08:22:32 (1448382152) Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ======== 08:22:33 (1448382153) Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ == 08:22:34 (1448382154) Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ == 08:22:35 (1448382155) Lustre: DEBUG MARKER: 32p_1 Lustre: DEBUG MARKER: 32p_2 Lustre: DEBUG MARKER: 32p_3 Lustre: DEBUG MARKER: 32p_4 Lustre: DEBUG MARKER: 32p_5 Lustre: DEBUG MARKER: 32p_6 Lustre: DEBUG MARKER: 32p_7 Lustre: DEBUG MARKER: 32p_8 Lustre: DEBUG MARKER: 32p_9 Lustre: DEBUG MARKER: 32p_10 Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) == 08:22:38 (1448382158) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) == 08:22:39 (1448382159) EXT2-fs warning (device loop0): ext2_fill_super: mounting ext3 filesystem as ext2 SELinux: initialized (dev loop0, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ====== 08:22:40 (1448382160) Lustre: DEBUG MARKER: 33_1 Lustre: DEBUG MARKER: 33_2 Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) == 08:22:41 (1448382161) Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) == 08:22:42 (1448382162) Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes == 08:22:43 (1448382163) Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir == 08:22:44 (1448382164) Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode == 08:22:45 (1448382165) Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory == 08:22:46 (1448382166) Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ============= 08:22:47 (1448382167) Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ======= 08:22:48 (1448382168) Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ================ 08:22:48 (1448382168) Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file =============================== 08:22:49 (1448382169) Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ==== 08:22:50 (1448382170) Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ========= 08:22:51 (1448382171) Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ================================= 08:22:52 (1448382172) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 08:22:52 (1448382172) Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ======= 08:22:56 (1448382176) Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ===================== 08:22:57 (1448382177) Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ====================== 08:22:58 (1448382178) Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ============ 08:22:58 (1448382178) Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ============= 08:22:59 (1448382179) Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ====== 08:23:00 (1448382180) Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ============ 08:23:01 (1448382181) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ======================= 08:23:03 (1448382183) Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ============ 08:23:46 (1448382226) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory == 08:23:48 (1448382228) Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ===== 08:23:49 (1448382229) Lustre: DEBUG MARKER: == sanity test 39: mtime changed on create ============================= 08:23:50 (1448382230) Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ======== 08:23:53 (1448382233) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ============================= 08:23:55 (1448382235) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ================================ 08:23:58 (1448382238) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ========================== 08:23:59 (1448382239) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =================== 08:23:59 (1448382239) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ================================= 08:24:02 (1448382242) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat =============== 08:24:05 (1448382245) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ================================ 08:24:07 (1448382247) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ========================= 08:24:09 (1448382249) LustreError: 4765:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 sleeping for 1000ms LustreError: 4765:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 awake Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ========================== 08:24:13 (1448382253) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ============================= 08:24:15 (1448382255) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 == 08:24:23 (1448382263) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored == 08:24:26 (1448382266) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ========== 08:24:37 (1448382277) Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========== 08:24:38 (1448382278) Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate =============== 08:24:39 (1448382279) Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ======================= 08:24:39 (1448382279) Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously == 08:24:42 (1448382282) Lustre: DEBUG MARKER: == sanity test 43: execution of file opened for write should return -ETXTBSY == 08:25:39 (1448382339) Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY == 08:25:41 (1448382341) Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY == 08:25:42 (1448382342) Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre========================== 08:25:43 (1448382343) Lustre: DEBUG MARKER: == sanity test 44: zero length read from a sparse stripe =============== 08:25:44 (1448382344) Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ================================= 08:25:45 (1448382345) Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================== 08:25:46 (1448382346) Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=== 08:25:47 (1448382347) Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=== 08:25:48 (1448382348) Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) == 08:25:49 (1448382349) Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) == 08:25:50 (1448382350) Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) == 08:25:51 (1448382351) Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent == 08:25:52 (1448382352) Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ================= 08:25:53 (1448382353) Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ==== 08:25:53 (1448382353) Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit == 08:25:56 (1448382356) Lustre: DEBUG MARKER: == sanity test 51ba: verify nlink for many subdirectory cleanup == 08:29:20 (1448382560) Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping ALWAYS excluded test 51d Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit == 08:32:26 (1448382746) Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ======= 08:47:56 (1448383676) Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ========= 08:47:57 (1448383677) Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ====== 08:47:58 (1448383678) Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ============================ 08:48:00 (1448383680) Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ======================== 08:48:01 (1448383681) Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ======================= 08:48:02 (1448383682) SELinux: initialized (dev loop3, type ext2), uses xattr Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ======================== 08:48:04 (1448383684) Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ======================== 08:48:05 (1448383685) Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe == 08:48:05 (1448383685) Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name =============================== 08:48:07 (1448383687) Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============================== 08:48:08 (1448383688) Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========= 08:48:09 (1448383689) Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============================== 08:48:10 (1448383690) Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============================== 08:48:11 (1448383691) Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============================== 08:48:12 (1448383692) Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============================== 08:48:14 (1448383694) Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============================== 08:48:15 (1448383695) Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================ 08:48:16 (1448383696) Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ================================= 08:48:18 (1448383698) Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ================================= 08:48:19 (1448383699) Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ============================ 08:48:21 (1448383701) Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works == 08:48:23 (1448383703) Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works == 08:48:24 (1448383704) Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works == 08:48:26 (1448383706) Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ========= 08:48:27 (1448383707) Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works == 08:48:29 (1448383709) Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support == 08:48:40 (1448383720) Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released == 08:48:41 (1448383721) Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error == 08:48:42 (1448383722) Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir == 08:48:44 (1448383724) Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ==== 08:48:45 (1448383725) Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ===== 08:48:46 (1448383726) Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ================ 08:48:52 (1448383732) Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =========== 08:48:52 (1448383732) Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============ 08:49:05 (1448383745) Lustre: DEBUG MARKER: test_60 run 8299 - from kernel mode LustreError: 11-0: lustre-MDT0000-mdc-ffff88007da03400: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ffff88007da03400: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 13029:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1448383767/real 1448383767] req@ffff88007c0933c0 x1518738112426548/t0(0) o400->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 224/224 e 0 to 1 dl 1448383774 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13029:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0x18d6cd0c95be2052 to 0x18d6cd0c964e123c Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 1 previous similar message LustreError: 13026:0:(client.c:2899:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88004c64a9c0 x1518738110414112/t12884927340(12884927340) o101->lustre-MDT0000-mdc-ffff88007da03400@10.100.4.183@tcp:12/10 lens 872/544 e 0 to 0 dl 1448383781 ref 2 fl Interpret:RP/4/0 rc 301/301 Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========== 08:49:35 (1448383775) Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full == 08:49:36 (1448383776) Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 08:50:35 (1448383835) Lustre: DEBUG MARKER: test message ID 26195 15602 Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created == 08:50:36 (1448383836) Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================== 08:50:37 (1448383837) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 08:50:38 (1448383838) Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ===== 08:51:39 (1448383899) Lustre: *** cfs_fail_loc=406, val=0*** LustreError: 3466:0:(osc_request.c:1721:osc_build_rpc()) prep_req failed: -12 LustreError: 3466:0:(osc_cache.c:2157:osc_check_rpcs()) Write request failed with -12 Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ======= 08:51:42 (1448383902) Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ == 08:51:44 (1448383904) Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ====================== 08:51:44 (1448383904) Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S 2097152 -i 0 -c 1 == 08:51:45 (1448383905) Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S 4194304 -i 1 -c 1 == 08:51:46 (1448383906) Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S 1048576 -c stripe_count == 08:51:47 (1448383907) Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========================= 08:51:48 (1448383908) Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ===== 08:51:49 (1448383909) Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d ============================= 08:51:49 (1448383909) Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ====================== 08:51:50 (1448383910) Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)=== 08:51:51 (1448383911) Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping == 08:51:52 (1448383912) Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping == 08:51:54 (1448383914) Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping == 08:51:56 (1448383916) Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=== 08:52:05 (1448383925) Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs == 08:52:07 (1448383927) Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ========================== 08:52:10 (1448383930) Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ================= 08:52:10 (1448383930) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: SKIP: sanity test_68a skipping SLOW test 68a (base 68) Lustre: DEBUG MARKER: SKIP: sanity test_68b skipping ALWAYS excluded test 68b Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ======== 08:52:13 (1448383933) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ====== 08:52:18 (1448383938) Lustre: DEBUG MARKER: cancel_lru_locks mdc start Lustre: DEBUG MARKER: cancel_lru_locks mdc stop Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) == 08:52:19 (1448383939) Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) == 08:52:21 (1448383941) Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) == 08:52:48 (1448383968) Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) == 08:52:48 (1448383968) Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) == 08:52:49 (1448383969) Lustre: *** cfs_fail_loc=319, val=0*** Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation == 08:52:50 (1448383970) Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read == 08:52:53 (1448383973) Lustre: *** cfs_fail_loc=409, val=0*** LustreError: 132-0: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.100.4.184@tcp inode [0x200000401:0xb523:0x0] object 0x0:5129 extent [0-1048575] LustreError: 13028:0:(osc_request.c:1233:check_write_checksum()) original client csum e4dc3a56 (type 2), server csum e4dc3a55 (type 2), client csum now e4dc3a55 LustreError: 13028:0:(osc_request.c:1396:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007d1a2080 x1518738112574028/t8589938472(8589938472) o4->lustre-OST0001-osc-ffff88007da03400@10.100.4.184@tcp:6/4 lens 608/416 e 0 to 0 dl 1448383980 ref 3 fl Interpret:R/4/0 rc 0/0 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: set checksum type to crc32 Lustre: *** cfs_fail_loc=408, val=0*** LustreError: 133-1: lustre-OST0001-osc-ffff88007da03400: BAD READ CHECKSUM: from 10.100.4.184@tcp inode [0x0:0x0:0x0] object 0x0:5129 extent [3145728-4194303] LustreError: 13030:0:(osc_request.c:1360:osc_brw_fini_request()) client 8d8ac0e9, server 1ff2c187, cksum_type 2 LustreError: 13030:0:(osc_request.c:1396:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880049de2080 x1518738112574136/t0(0) o3->lustre-OST0001-osc-ffff88007da03400@10.100.4.184@tcp:6/4 lens 608/400 e 0 to 0 dl 1448383982 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: set checksum type to adler LustreError: 133-1: lustre-OST0001-osc-ffff88007da03400: BAD READ CHECKSUM: from 10.100.4.184@tcp inode [0x0:0x0:0x0] object 0x0:5129 extent [1048576-2097151] LustreError: 13030:0:(osc_request.c:1360:osc_brw_fini_request()) client 4a727d07, server 8ff47e65, cksum_type 2 LustreError: 13030:0:(osc_request.c:1396:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880037462680 x1518738112574212/t0(0) o3->lustre-OST0001-osc-ffff88007da03400@10.100.4.184@tcp:6/4 lens 608/400 e 0 to 0 dl 1448383984 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Lustre: DEBUG MARKER: set checksum type to adler Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read == 08:52:59 (1448383979) Lustre: *** cfs_fail_loc=409, val=0*** Lustre: Skipped 1 previous similar message LustreError: 132-0: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 10.100.4.184@tcp inode [0x200000401:0xb524:0x0] object 0x0:4268 extent [0-1048575] LustreError: 13027:0:(osc_request.c:1233:check_write_checksum()) original client csum 19eeae62 (type 2), server csum 19eeae61 (type 2), client csum now 19eeae61 LustreError: 13027:0:(osc_request.c:1396:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88006e1af080 x1518738112574300/t12884907280(12884907280) o4->lustre-OST0000-osc-ffff88007da03400@10.100.4.184@tcp:6/4 lens 608/416 e 0 to 0 dl 1448383986 ref 2 fl Interpret:RM/0/0 rc 0/0 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop LustreError: 133-1: lustre-OST0000-osc-ffff88007da03400: BAD READ CHECKSUM: from 10.100.4.184@tcp inode [0x0:0x0:0x0] object 0x0:4268 extent [1048576-2097151] LustreError: 13027:0:(osc_request.c:1360:osc_brw_fini_request()) client a570acd1, server 19eeae61, cksum_type 2 Lustre: DEBUG MARKER: SKIP: sanity test_77f skipping SLOW test 77f Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read == 08:53:03 (1448383983) LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.100.4.184@tcp inode [0x200000401:0xb526:0x0] object 0x0:4269 extent [0-1048575] LustreError: 13028:0:(osc_request.c:1233:check_write_checksum()) original client csum e4dc3a55 (type 2), server csum 180f3a8c (type 2), client csum now e4dc3a55 LustreError: 13028:0:(osc_request.c:1396:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007c19f3c0 x1518738112574472/t12884907289(12884907289) o4->lustre-OST0000-osc-ffff88007da03400@10.100.4.184@tcp:6/4 lens 608/416 e 0 to 0 dl 1448383991 ref 3 fl Interpret:R/4/0 rc 0/0 LustreError: 13028:0:(osc_request.c:1396:osc_brw_redo_request()) Skipped 1 previous similar message Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop LustreError: 133-1: lustre-OST0000-osc-ffff88007da03400: BAD READ CHECKSUM: from 10.100.4.184@tcp inode [0x0:0x0:0x0] object 0x0:4269 extent [1048576-2097151] LustreError: 13029:0:(osc_request.c:1360:osc_brw_fini_request()) client 4d397d0a, server 8ff47e65, cksum_type 2 Lustre: DEBUG MARKER: == sanity test 77i: client not supporting OSD_CONNECT_CKSUM == 08:53:08 (1448383988) Lustre: Unmounted lustre-client Lustre: *** cfs_fail_loc=40b, val=0*** Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 == 08:53:10 (1448383990) Lustre: Unmounted lustre-client Lustre: Skipped 1 previous similar message Lustre: *** cfs_fail_loc=40c, val=0*** Lustre: Mounted lustre-client Lustre: Skipped 1 previous similar message SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: SKIP: sanity test_78 skipping SLOW test 78 Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ========================= 08:53:14 (1448383994) Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ====== 08:53:16 (1448383996) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ================= 08:53:18 (1448383998) Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ========= 08:53:19 (1448383999) Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ================================= 08:53:20 (1448384000) Lustre: DEBUG MARKER: == sanity test 99a: cvs init =========================================== 08:53:24 (1448384004) Lustre: DEBUG MARKER: == sanity test 99b: cvs import ========================================= 08:53:26 (1448384006) Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ======================================= 08:53:28 (1448384008) Lustre: DEBUG MARKER: == sanity test 99d: cvs add ============================================ 08:53:34 (1448384014) Lustre: DEBUG MARKER: == sanity test 99e: cvs update ========================================= 08:53:35 (1448384015) Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ========================================= 08:53:36 (1448384016) Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ============= 08:53:38 (1448384018) Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads ================== 08:53:40 (1448384020) Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =================== 08:54:32 (1448384072) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead =================== 08:54:39 (1448384079) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled == 08:55:04 (1448384104) Lustre: DEBUG MARKER: SKIP: sanity test_101d Need free space 500M, have 469M Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) == 08:55:05 (1448384105) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 101f: check read-ahead for max_read_ahead_whole_mb == 08:55:25 (1448384125) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ==================================== 08:55:26 (1448384126) Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ============== 08:55:27 (1448384127) Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ============= 08:55:28 (1448384128) Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============= 08:55:29 (1448384129) Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ============= 08:55:32 (1448384132) Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block == 08:55:35 (1448384135) Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode == 08:55:37 (1448384137) Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ============== 08:55:38 (1448384138) Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ===== 08:55:39 (1448384139) Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash == 08:55:42 (1448384142) Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ==================================== 08:55:42 (1448384142) Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========== 08:55:43 (1448384143) Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs == 08:55:44 (1448384144) Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission == 08:55:45 (1448384145) Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans == 08:55:46 (1448384146) Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values == 08:55:47 (1448384147) Lustre: DEBUG MARKER: == sanity test 103a: acl test =========================================== 08:55:48 (1448384148) Lustre: DEBUG MARKER: sanity test_103a: @@@@@@ FAIL: run_acl_subtest cp failed