Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: 9539:0:(llite_lib.c:2628:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.100.4.183@tcp:/lustre/fid: [0x20004bef3:0x38:0x0]//d24b.recovery-small/f24b.recovery-small-2 may get corrupted (rc -108) Lustre: DEBUG MARKER: SKIP: recovery-small test_26a skipping SLOW test 26a Lustre: DEBUG MARKER: SKIP: recovery-small test_26b skipping SLOW test 26b Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's == 22:14:51 (1450217691) LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d26ec00: operation mds_reint to node 10.100.4.183@tcp failed: rc = -19 LustreError: Skipped 8 previous similar messages LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d572a485e to 0xa5e82e9d5737c587 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: == recovery-small test 28: handle error adding new clients (bug 6086) == 22:17:48 (1450217868) Lustre: *** cfs_fail_loc=305, val=0*** Lustre: Skipped 106 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) == 22:18:22 (1450217902) LustreError: 167-0: lustre-MDT0000-mdc-ffff88007d26ec00: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message LustreError: 24355:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007d26ec00), error -5 LustreError: 24355:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) == 22:18:51 (1450217931) Lustre: DEBUG MARKER: SKIP: recovery-small test_50 skipping SLOW test 50 Lustre: DEBUG MARKER: SKIP: recovery-small test_51 skipping SLOW test 51 Lustre: DEBUG MARKER: SKIP: recovery-small test_52 skipping ALWAYS excluded test 52 Lustre: DEBUG MARKER: == recovery-small test 53: touch: drop rep == 22:19:00 (1450217940) Lustre: DEBUG MARKER: == recovery-small test 54: back in time == 22:19:01 (1450217941) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: == recovery-small test 55: ost_brw_read/write drops timed-out read/write request == 22:19:40 (1450217980) Lustre: DEBUG MARKER: == recovery-small test 56: do not fail on getattr resend == 22:21:03 (1450218063) Lustre: DEBUG MARKER: SKIP: recovery-small test_57 skipping SLOW test 57 Lustre: DEBUG MARKER: == recovery-small test 58: Eviction in the middle of open RPC reply processing == 22:24:24 (1450218264) LustreError: 28308:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 sleeping for 100000ms Lustre: *** cfs_fail_loc=305, val=0*** LustreError: 28308:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id 801 awake Lustre: DEBUG MARKER: == recovery-small test 59: Read cancel race on client eviction == 22:26:05 (1450218365) Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: SKIP: recovery-small test_60 skipping ALWAYS excluded test 60 Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 == 22:26:07 (1450218367) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88007d26ec00: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 38 previous similar messages LustreError: 167-0: lustre-MDT0000-mdc-ffff88007d26ec00: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message LustreError: 29753:0:(lmv_obd.c:1465:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007d26ec00), error -5 Lustre: lustre-MDT0000-mdc-ffff88007d26ec00: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 43 previous similar messages LustreError: 29753:0:(llite_lib.c:1752:ll_statfs_internal()) md_statfs fails: rc = -5 Lustre: DEBUG MARKER: == recovery-small test 66: lock enqueue re-send vs client eviction == 22:26:21 (1450218381) Lustre: DEBUG MARKER: == recovery-small test 65: lock enqueue for destroyed export == 22:26:27 (1450218387) LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 2 previous similar messages LustreError: 15c-8: MGC10.100.4.183@tcp: The configuration from log 'lustre-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d5737f7c9 to 0xa5e82e9d573802d5 Lustre: Skipped 2 previous similar messages Lustre: Unmounted lustre-client LustreError: 30827:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-5) Lustre: DEBUG MARKER: recovery-small test_65: @@@@@@ FAIL: test_65 failed with 1 Lustre: DEBUG MARKER: -----============= acceptance-small: replay-ost-single ============----- Tue Dec 15 22:26:29 UTC 2015 Lustre: DEBUG MARKER: excepting tests: Lustre: DEBUG MARKER: skipping tests SLOW=no: 5 Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: == replay-ost-single test 0a: target handle mismatch (bug 5317) == 22:26:35 (1450218395) Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 2 previous similar messages Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1450218396/real 1450218396] req@ffff8800796b86c0 x1520653181827792/t0(0) o8->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:28/4 lens 520/544 e 0 to 1 dl 1450218401 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 108 previous similar messages Lustre: DEBUG MARKER: == replay-ost-single test 0b: empty replay == 22:27:02 (1450218422) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-ffff88007d14b000: operation ost_statfs to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 9 previous similar messages Lustre: DEBUG MARKER: == replay-ost-single test 1: touch == 22:27:22 (1450218442) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == replay-ost-single test 2: |x| 10 open(O_CREAT)s == 22:27:40 (1450218460) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification == 22:28:05 (1450218485) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: == replay-ost-single test 4: Fail OST during read, with verification == 22:28:29 (1450218509) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 8 sec Lustre: DEBUG MARKER: SKIP: replay-ost-single test_5 skipping SLOW test 5 Lustre: DEBUG MARKER: == replay-ost-single test 6: Fail OST before obd_destroy == 22:28:54 (1450218534) Lustre: DEBUG MARKER: before: 4879812 after_dd: 4874692 took 0 seconds Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: before: 4879812 after: 4879812 Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy == 22:29:40 (1450218580) Lustre: DEBUG MARKER: before: 4879812 after_dd: 4874692 took 0 seconds Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: before: 4879812 after: 4879812 Lustre: DEBUG MARKER: == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error == 22:30:25 (1450218625) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == replay-ost-single test 8b: Verify redo io: redo io should success after recovery == 22:33:03 (1450218783) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction == 22:36:02 (1450218962) LustreError: 11-0: lustre-OST0000-osc-ffff88007d14b000: operation ost_write to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 7 previous similar messages Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection to lustre-OST0000 (at 10.100.4.184@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages LustreError: 167-0: lustre-OST0000-osc-ffff88007d14b000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message LustreError: 10438:0:(ldlm_resource.c:887:ldlm_resource_complain()) lustre-OST0000-osc-ffff88007d14b000: namespace resource [0x8792:0x0:0x0].0x0 (ffff8800577aed40) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 10438:0:(ldlm_resource.c:887:ldlm_resource_complain()) Skipped 1 previous similar message LustreError: 10438:0:(ldlm_resource.c:1502:ldlm_resource_dump()) --- Resource: [0x8792:0x0:0x0].0x0 (ffff8800577aed40) refcount = 2 LustreError: 10438:0:(ldlm_resource.c:1505:ldlm_resource_dump()) Granted locks (in reverse order): LustreError: 10438:0:(ldlm_resource.c:1508:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff88007d14b000 lock: ffff88007abdd700/0x613a88617de6060d lrc: 4/0,1 mode: PW/PW res: [0x8792:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x126400000000 nid: local remote: 0x79dd433e0c4d9b78 expref: -99 pid: 10417 timeout: 0 lvb_type: 1 LustreError: 10438:0:(ldlm_resource.c:1502:ldlm_resource_dump()) --- Resource: [0x8792:0x0:0x0].0x0 (ffff8800577aed40) refcount = 1 Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 12 previous similar messages Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS == 22:39:26 (1450219166) Lustre: DEBUG MARKER: == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS == 22:42:57 (1450219377) Lustre: DEBUG MARKER: == replay-ost-single test 9: Verify that no req deadline happened during recovery == 22:44:41 (1450219481) Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1450219498/real 1450219498] req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 1 to 1 dl 1450219500 ref 2 fl Rpc:X/4/ffffffff rc 0/-1 Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1 previous similar message LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 1 to 1 dl 1450219500 ref 2 fl Interpret:EX/4/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 1 to 1 dl 1450219511 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 1 to 1 dl 1450219518 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 2 to 1 dl 1450219520 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 2 to 1 dl 1450219532 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 2 to 1 dl 1450219546 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) Skipped 1 previous similar message LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 3 to 1 dl 1450219566 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) Skipped 2 previous similar messages LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aac80 x1520653181861528/t188978562307(188978562307) o10->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 560/400 e 5 to 1 dl 1450219595 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) Skipped 4 previous similar messages LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) @@@ request replay timed out. req@ffff8800576aa980 x1520653181861536/t188978562308(188978562308) o4->lustre-OST0000-osc-ffff88007d14b000@10.100.4.184@tcp:6/4 lens 608/416 e 3 to 1 dl 1450219669 ref 2 fl Interpret:EX/6/ffffffff rc -110/-1 LustreError: 9538:0:(client.c:2853:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: DEBUG MARKER: == replay-ost-single test complete, duration 1413 sec == 22:50:02 (1450219802) Lustre: DEBUG MARKER: -----============= acceptance-small: insanity ============----- Tue Dec 15 22:50:03 UTC 2015 Lustre: DEBUG MARKER: excepting tests: 15 Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: == insanity test 0: Fail all nodes, independently == 22:50:08 (1450219808) Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d14b000: operation mds_statfs to node 10.100.4.183@tcp failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ffff88007d14b000: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d57380c8a to 0xa5e82e9d57382183 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == insanity test 1: MDS/MDS failure == 22:51:07 (1450219867) Lustre: DEBUG MARKER: SKIP: insanity test_1 needs >= 2 MDTs Lustre: DEBUG MARKER: == insanity test 2: Second Failure Mode: MDS/OST Tue Dec 15 22:51:08 UTC 2015 == 22:51:08 (1450219868) LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1450220111/real 1450220111] req@ffff88007c1936c0 x1520653181862188/t0(0) o38->lustre-MDT0000-mdc-ffff88007d14b000@10.100.4.183@tcp:12/10 lens 520/544 e 0 to 1 dl 1450220147 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 43 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d57382183 to 0xa5e82e9d573823f2 Lustre: DEBUG MARKER: == insanity test 3: Thirdb Failure Mode: MDS/CLIENT Tue Dec 15 22:58:58 UTC 2015 == 22:58:58 (1450220338) LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: == insanity test 4: Fourth Failure Mode: OST/MDS Tue Dec 15 22:59:26 UTC 2015 == 22:59:26 (1450220366) LustreError: 445:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 LustreError: 445:0:(mgc_request.c:527:do_requeue()) failed processing log: -5 Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1450220701/real 1450220701] req@ffff88007af98c80 x1520653181862600/t0(0) o250->MGC10.100.4.183@tcp@10.100.4.183@tcp:26/25 lens 520/544 e 0 to 1 dl 1450220756 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 9538:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d573823f2 to 0xa5e82e9d57382725 Lustre: MGC10.100.4.183@tcp: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: == insanity test 5: Fifth Failure Mode: OST/OST Tue Dec 15 23:07:45 UTC 2015 == 23:07:45 (1450220865) LustreError: 11-0: lustre-OST0000-osc-ffff88007d14b000: operation ost_statfs to node 10.100.4.184@tcp failed: rc = -107 LustreError: Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection to lustre-OST0000 (at 10.100.4.184@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: == insanity test 6: Sixth Failure Mode: OST/CLIENT Tue Dec 15 23:10:03 UTC 2015 == 23:10:03 (1450221003) Lustre: DEBUG MARKER: Request fail clients: , to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: == insanity test 7: Seventh Failure Mode: CLIENT/MDS Tue Dec 15 23:10:45 UTC 2015 == 23:10:45 (1450221045) Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d57382725 to 0xa5e82e9d57382c0a Lustre: DEBUG MARKER: == insanity test 8: Eighth Failure Mode: CLIENT/OST Tue Dec 15 23:13:13 UTC 2015 == 23:13:13 (1450221193) Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: == insanity test 9: Ninth Failure Mode: CLIENT/CLIENT Tue Dec 15 23:15:42 UTC 2015 == 23:15:42 (1450221342) Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 0, failed: 0 Lustre: DEBUG MARKER: No clients failed! Lustre: DEBUG MARKER: == insanity test 10: Tenth Failure Mode: MDT0/OST/MDT1 Tue Dec 15 23:17:50 UTC 2015 == 23:17:50 (1450221470) Lustre: DEBUG MARKER: SKIP: insanity test_10 needs >= 2 MDTs Lustre: DEBUG MARKER: == insanity test 11: Eleventh Failure Mode: MDS0/CLIENT/MDS1 Tue Dec 15 23:17:51 UTC 2015 == 23:17:51 (1450221471) Lustre: DEBUG MARKER: SKIP: insanity test_11 needs >= 2 MDTs Lustre: DEBUG MARKER: == insanity test 12: Twelve Failure Mode: MDS0,MDS1/OST0, OST1/CLIENTS Tue Dec 15 23:17:52 UTC 2015 == 23:17:52 (1450221472) Lustre: DEBUG MARKER: SKIP: insanity test_12 needs >= 2 MDTs Lustre: DEBUG MARKER: == insanity test 13: Thirteen Failure Mode: MDS0,MDS1/CLIENTS/OST0,OST1 Tue Dec 15 23:17:53 UTC 2015 == 23:17:53 (1450221473) Lustre: DEBUG MARKER: SKIP: insanity test_13 needs >= 2 MDTs Lustre: DEBUG MARKER: == insanity test 14: Fourteen Failure Mode: OST0,OST1/CLIENTS/MDS0,MDS1 Tue Dec 15 23:17:54 UTC 2015 == 23:17:54 (1450221474) Lustre: DEBUG MARKER: SKIP: insanity test_14 needs >= 2 MDTs Lustre: DEBUG MARKER: SKIP: insanity test_15 skipping ALWAYS excluded test 15 Lustre: DEBUG MARKER: == insanity test complete, duration 1673 sec == 23:17:56 (1450221476) Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Tue Dec 15 23:17:56 UTC 2015 Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: excepting tests: Lustre: DEBUG MARKER: skipping tests SLOW=no: 9 18 21 Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance == 23:18:07 (1450221487) Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) == 23:18:36 (1450221516) Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) Lustre: DEBUG MARKER: Write... Lustre: DEBUG MARKER: Write out of block quota ... Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: -------------------------------------- Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) Lustre: DEBUG MARKER: Write ... Lustre: DEBUG MARKER: Write out of block quota ... Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 2: File hard limit (normal use and out of quota) == 23:19:07 (1450221547) Lustre: DEBUG MARKER: User quota (inode hardlimit:1024 files) Lustre: DEBUG MARKER: Create 1024 files ... Lustre: DEBUG MARKER: Create out of file quota ... Lustre: DEBUG MARKER: -------------------------------------- Lustre: DEBUG MARKER: Group quota (inode hardlimit:1024 files) Lustre: DEBUG MARKER: Create 1024 files ... Lustre: DEBUG MARKER: Create out of file quota ... Lustre: DEBUG MARKER: == sanity-quota test 3: Block soft limit (start timer, timer goes off, stop timer) == 23:19:36 (1450221576) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 4a: File soft limit (start timer, timer goes off, stop timer) == 23:21:29 (1450221689) Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling == 23:23:02 (1450221782) Lustre: DEBUG MARKER: == sanity-quota test 5: Chown Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master == 23:23:20 (1450221800) Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) == 23:27:39 (1450222059) Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection to lustre-OST0000 (at 10.100.4.184@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) == 23:29:22 (1450222162) Lustre: lustre-OST0000-osc-ffff88007d14b000: Connection restored to 10.100.4.184@tcp (at 10.100.4.184@tcp) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) == 23:29:57 (1450222197) LustreError: 11-0: lustre-MDT0000-mdc-ffff88007d14b000: operation obd_ping to node 10.100.4.183@tcp failed: rc = -107 LustreError: 166-1: MGC10.100.4.183@tcp: Connection to MGS (at 10.100.4.183@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: Evicted from MGS (at 10.100.4.183@tcp) after server handle changed from 0xa5e82e9d57382c0a to 0xa5e82e9d573a920b Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) == 23:33:02 (1450222382) Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) == 23:35:29 (1450222529) Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e Required more MDTs Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled == 23:35:30 (1450222530) Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9 Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user == 23:38:19 (1450222699) LustreError: 23481:0:(mdc_request.c:1959:mdc_quotactl()) ptlrpc_queue_wait failed, rc: -1 Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota == 23:38:32 (1450222712) Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing == 23:38:42 (1450222722) Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing == 23:40:13 (1450222813) Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b skipping rebalancing test Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list == 23:40:14 (1450222814) Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota == 23:40:27 (1450222827) Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error == 23:40:32 (1450222832) Lustre: DEBUG MARKER: SKIP: sanity-quota test_18 skipping SLOW test 18 Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) == 23:42:03 (1450222923) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) == 23:42:17 (1450222937) Lustre: DEBUG MARKER: SKIP: sanity-quota test_21 skipping SLOW test 21 Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param' == 23:42:21 (1450222941) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: Unmounted lustre-client Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) == 23:46:09 (1450223169) Lustre: DEBUG MARKER: OST0_SIZE: 2484420 required: 6144 Lustre: DEBUG MARKER: run for 4MB test file Lustre: DEBUG MARKER: User quota (limit: 4 MB) Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT Lustre: DEBUG MARKER: Write half of file Lustre: DEBUG MARKER: Write out of block quota ... Lustre: DEBUG MARKER: Step1: done Lustre: DEBUG MARKER: Step2: rewrite should succeed Lustre: DEBUG MARKER: Step2: done Lustre: DEBUG MARKER: OST0_SIZE: 2484420 required: 61440 Lustre: DEBUG MARKER: run for 40MB test file Lustre: DEBUG MARKER: User quota (limit: 40 MB) Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT Lustre: DEBUG MARKER: Write half of file Lustre: DEBUG MARKER: Write out of block quota ... Lustre: DEBUG MARKER: Step1: done Lustre: DEBUG MARKER: Step2: rewrite should succeed Lustre: DEBUG MARKER: Step2: done Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) == 23:46:57 (1450223217) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) == 23:47:12 (1450223232) Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group ID (b20200) == 23:47:13 (1450223233) Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output == 23:47:16 (1450223236) Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times == 23:47:18 (1450223238) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot == 23:48:51 (1450223331) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: Restart... Lustre: Unmounted lustre-client Lustre: 18322:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism LNet: Removed LNI 10.100.4.186@tcp LNet: HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6.6,ib_stack=inkernel-3268-gf045898-PRISTINE-2.6.32-504.30.3.el6.x86_64 LNet: Added LNI 10.100.4.186@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: 18821:0:(gss_svc_upcall.c:1129:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active Lustre: 18821:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: Mounted lustre-client SELinux: initialized (dev lustre, type lustre), uses xattr Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' == 23:50:34 (1450223434) Lustre: DEBUG MARKER: == sanity-quota test complete, duration 1984 sec == 23:51:00 (1450223460) Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Dec 15 23:51:00 UTC 2015 Lustre: DEBUG MARKER: Using TIMEOUT=100 Lustre: DEBUG MARKER: excepting tests: 76 42a 42b 42c 42d 45 51d 68b Lustre: DEBUG MARKER: skipping tests SLOW=no: 24o 24D 27m 64b 68 71 77f 78 115 124b 300o Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ======================= 23:51:06 (1450223466) Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre =============================== 23:51:07 (1450223467) Lustre: DEBUG MARKER: == sanity test 0c: check import proc =============================== 23:51:07 (1450223467) Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================================ 23:51:08 (1450223468) Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ====================== 23:51:09 (1450223469) Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ======================= 23:51:10 (1450223470) Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) == 23:51:11 (1450223471) Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============== 23:51:12 (1450223472) Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ==== 23:51:12 (1450223472) Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ==== 23:51:13 (1450223473) Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ==== 23:51:14 (1450223474) Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? == 23:51:15 (1450223475) Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) == 23:51:16 (1450223476) Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ====== 23:51:17 (1450223477) Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ======= 23:51:18 (1450223478) Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ========= 23:51:18 (1450223478) Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================== 23:51:19 (1450223479) Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ======== 23:51:20 (1450223480) Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============== 23:51:21 (1450223481) Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ========== 23:51:22 (1450223482) Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ========== 23:51:23 (1450223483) Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ========= 23:51:23 (1450223483) Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ============ 23:51:24 (1450223484) Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f =================== 23:51:25 (1450223485) Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ==================== 23:51:26 (1450223486) Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ================ 23:51:27 (1450223487) Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ====== 23:51:28 (1450223488) Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ========================== 23:51:28 (1450223488) Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ====== 23:51:29 (1450223489) Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================== 23:51:30 (1450223490) Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries == 23:51:31 (1450223491) Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug == 23:51:33 (1450223493) Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink == 23:51:34 (1450223494) LustreError: 3782:0:(symlink.c:98:ll_readlink_internal()) lustre: inode [0x200053bf1:0x52:0x0]: symlink length 33 not expected 35 Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled =========================== 23:51:35 (1450223495) Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========== 23:51:36 (1450223496) Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink == 23:51:37 (1450223497) LustreError: 11-0: lustre-MDT0000-mdc-ffff880057401c00: operation ldlm_enqueue to node 10.100.4.183@tcp failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff880057401c00: Connection to lustre-MDT0000 (at 10.100.4.183@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 18808:0:(client.c:2908:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007aa0b3c0 x1520669433007152/t730144440594(730144440594) o101->lustre-MDT0000-mdc-ffff880057401c00@10.100.4.183@tcp:12/10 lens 872/544 e 0 to 0 dl 1450223533 ref 2 fl Interpret:RP/4/0 rc 301/301 Lustre: lustre-MDT0000-mdc-ffff880057401c00: Connection restored to 10.100.4.183@tcp (at 10.100.4.183@tcp) Lustre: DEBUG MARKER: sanity test_17m: @@@@@@ FAIL: e2fsck should not report error upon short/long symlink MDT: rc=4