[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.10.1.el7_lustre.x86_64 (jenkins@trevis-306-el7-x8664-4.trevis.whamcloud.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.10.1.el7_lustre.x86_64 root=UUID=4fc0d716-02d1-4c3d-9da0-ae24f012d5df ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fff9fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffa000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x7fffa max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000080000000 mask 3FFF80000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] found SMP MP-table at [mem 0x000f72e0-0x000f72ef] mapped at [ffffffffff2002e0] [ 0.000000] Base memory trampoline at [ffff8c09c0099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x2a052000, 0x2a052fff] PGTABLE [ 0.000000] BRK [0x2a053000, 0x2a053fff] PGTABLE [ 0.000000] BRK [0x2a054000, 0x2a054fff] PGTABLE [ 0.000000] BRK [0x2a055000, 0x2a055fff] PGTABLE [ 0.000000] BRK [0x2a056000, 0x2a056fff] PGTABLE [ 0.000000] BRK [0x2a057000, 0x2a057fff] PGTABLE [ 0.000000] RAMDISK: [mem 0x34d23000-0x36689fff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f7110 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007ffffb04 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffff177 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffe040 01137 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 000000007fffe000 00040 [ 0.000000] ACPI: SSDT 000000007ffff1eb 00899 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007ffffa84 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fff9fff] [ 0.000000] NODE_DATA(0) allocated [mem 0x7ffd3000-0x7fff9fff] [ 0.000000] Reserving 161MB of memory at 480MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: cpu 0, msr 0:7ff82001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1383491923378 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0x7fff9fff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x7fff9fff] [ 0.000000] On node 0 totalpages: 524184 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 8128 pages used for memmap [ 0.000000] DMA32 zone: 520186 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8c0a3fc00000 s118784 r8192 d28672 u1048576 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc1c040 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515971 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.10.1.el7_lustre.x86_64 root=UUID=4fc0d716-02d1-4c3d-9da0-ae24f012d5df ro crashkernel=auto console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1853156k/2097128k available (7668k kernel code, 392k absent, 243580k reserved, 6052k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.748 MHz processor [ 0.098371] Calibrating delay loop (skipped) preset value.. 5187.49 BogoMIPS (lpj=2593748) [ 0.099315] pid_max: default: 32768 minimum: 301 [ 0.099883] Security Framework initialized [ 0.100371] SELinux: Initializing. [ 0.100796] SELinux: Starting in permissive mode [ 0.100798] Yama: becoming mindful. [ 0.101380] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.103136] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.104366] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes) [ 0.105110] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes) [ 0.106120] Initializing cgroup subsys memory [ 0.106614] Initializing cgroup subsys devices [ 0.107115] Initializing cgroup subsys freezer [ 0.107609] Initializing cgroup subsys net_cls [ 0.108107] Initializing cgroup subsys blkio [ 0.108580] Initializing cgroup subsys perf_event [ 0.109104] Initializing cgroup subsys hugetlb [ 0.109594] Initializing cgroup subsys pids [ 0.110067] Initializing cgroup subsys net_prio [ 0.110633] mce: CPU supports 10 MCE banks [ 0.111147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.111741] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.112324] tlb_flushall_shift: 6 [ 0.112719] Speculative Store Bypass: Vulnerable [ 0.113237] FEATURE SPEC_CTRL Not Present [ 0.113680] FEATURE IBPB_SUPPORT Not Present [ 0.114617] Spectre V2 : Vulnerable: Retpoline without IBPB [ 0.115674] Freeing SMP alternatives: 28k freed [ 0.123417] ACPI: Core revision 20130517 [ 0.124587] ACPI: All ACPI Tables successfully acquired [ 0.125299] ftrace: allocating 29195 entries in 115 pages [ 0.169956] Enabling x2apic [ 0.170313] Enabled x2apic [ 0.170858] Switched APIC routing to physical x2apic. [ 0.172479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.173275] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.174502] TSC deadline timer enabled [ 0.174513] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.175715] KVM setup paravirtual spinlock [ 0.177809] smpboot: Booting Node 0, Processors #1 OK [ 0.178746] kvm-clock: cpu 1, msr 0:7ff82041, secondary cpu clock [ 0.199668] Brought up 2 CPUs [ 0.199672] KVM setup async PF for cpu 1 [ 0.199676] kvm-stealtime: cpu 1, msr 7fd1c040 [ 0.200948] smpboot: Max logical packages: 2 [ 0.201412] smpboot: Total of 2 processors activated (10374.99 BogoMIPS) [ 0.202820] devtmpfs: initialized [ 0.205343] EVM: security.selinux [ 0.205734] EVM: security.ima [ 0.206080] EVM: security.capability [ 0.207820] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.208578] pinctrl core: initialized pinctrl subsystem [ 0.209240] RTC time: 23:26:40, date: 04/30/19 [ 0.209855] NET: Registered protocol family 16 [ 0.210650] ACPI: bus type PCI registered [ 0.211129] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211927] PCI: Using configuration type 1 for base access [ 0.217056] ACPI: Added _OSI(Module Device) [ 0.217535] ACPI: Added _OSI(Processor Device) [ 0.218037] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218542] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219126] ACPI: Added _OSI(Linux-Dell-Video) [ 0.220107] ACPI: EC: Look up EC in DSDT [ 0.221452] ACPI: Interpreter enabled [ 0.221884] ACPI: (supports S0 S5) [ 0.222295] ACPI: Using IOAPIC for interrupt routing [ 0.222843] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224114] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.227318] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228009] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.228734] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.229454] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230991] acpiphp: Slot [3] registered [ 0.231460] acpiphp: Slot [4] registered [ 0.231944] acpiphp: Slot [5] registered [ 0.232405] acpiphp: Slot [6] registered [ 0.232865] acpiphp: Slot [7] registered [ 0.233347] acpiphp: Slot [8] registered [ 0.233808] acpiphp: Slot [9] registered [ 0.234275] acpiphp: Slot [10] registered [ 0.234763] acpiphp: Slot [11] registered [ 0.235240] acpiphp: Slot [12] registered [ 0.235708] acpiphp: Slot [13] registered [ 0.236199] acpiphp: Slot [14] registered [ 0.236683] acpiphp: Slot [15] registered [ 0.237175] acpiphp: Slot [16] registered [ 0.237653] acpiphp: Slot [17] registered [ 0.238129] acpiphp: Slot [18] registered [ 0.238610] acpiphp: Slot [19] registered [ 0.239087] acpiphp: Slot [20] registered [ 0.239557] acpiphp: Slot [21] registered [ 0.240048] acpiphp: Slot [22] registered [ 0.240518] acpiphp: Slot [23] registered [ 0.240995] acpiphp: Slot [24] registered [ 0.241480] acpiphp: Slot [25] registered [ 0.241954] acpiphp: Slot [26] registered [ 0.242422] acpiphp: Slot [27] registered [ 0.242904] acpiphp: Slot [28] registered [ 0.243379] acpiphp: Slot [29] registered [ 0.243847] acpiphp: Slot [30] registered [ 0.244341] acpiphp: Slot [31] registered [ 0.244789] PCI host bridge to bus 0000:00 [ 0.245239] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245962] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246677] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.247465] pci_bus 0000:00: root bus resource [mem 0x80000000-0xfebfffff window] [ 0.248247] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248870] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249209] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.249672] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.255631] pci 0000:00:01.1: reg 0x20: [io 0xc180-0xc18f] [ 0.258009] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.258779] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.259470] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.260224] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.261060] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 [ 0.266972] pci 0000:00:01.2: reg 0x20: [io 0xc140-0xc15f] [ 0.269487] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.269823] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.270601] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271569] pci 0000:00:03.0: [10ec:8139] type 00 class 0x020000 [ 0.273870] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc0ff] [ 0.276148] pci 0000:00:03.0: reg 0x14: [mem 0xfebc0000-0xfebc00ff] [ 0.287498] pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.287726] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 [ 0.290071] pci 0000:00:04.0: reg 0x10: [io 0xc100-0xc13f] [ 0.292361] pci 0000:00:04.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.304126] pci 0000:00:05.0: [1af4:1002] type 00 class 0x00ff00 [ 0.305325] pci 0000:00:05.0: reg 0x10: [io 0xc160-0xc17f] [ 0.312920] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.313845] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.314748] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.315651] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.316522] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.317447] vgaarb: loaded [ 0.317870] SCSI subsystem initialized [ 0.318333] ACPI: bus type USB registered [ 0.318809] usbcore: registered new interface driver usbfs [ 0.319413] usbcore: registered new interface driver hub [ 0.320015] usbcore: registered new device driver usb [ 0.320671] EDAC MC: Ver: 3.0.0 [ 0.321295] PCI: Using ACPI for IRQ routing [ 0.321759] PCI: pci_cache_line_size set to 64 bytes [ 0.321864] e820: reserve RAM buffer [mem 0x0009f800-0x0009ffff] [ 0.321867] e820: reserve RAM buffer [mem 0x7fffa000-0x7fffffff] [ 0.322022] NetLabel: Initializing [ 0.322420] NetLabel: domain hash size = 128 [ 0.322887] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.323468] NetLabel: unlabeled traffic allowed by default [ 0.324170] amd_nb: Cannot enumerate AMD northbridges [ 0.324748] Switched to clocksource kvm-clock [ 0.333224] pnp: PnP ACPI init [ 0.333612] ACPI: bus type PNP registered [ 0.334140] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.334195] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.334243] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.334285] pnp 00:03: [dma 2] [ 0.334308] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.334398] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.334570] pnp: PnP ACPI: found 5 devices [ 0.335037] ACPI: bus type PNP unregistered [ 0.342976] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.342979] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.342981] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.342983] pci_bus 0000:00: resource 7 [mem 0x80000000-0xfebfffff window] [ 0.343016] NET: Registered protocol family 2 [ 0.343722] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.344589] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.345412] TCP: Hash tables configured (established 16384 bind 16384) [ 0.346191] TCP: reno registered [ 0.346560] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.347253] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.348009] NET: Registered protocol family 1 [ 0.348502] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.349163] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.349806] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.350777] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.352179] PCI: CLS 0 bytes, default 64 [ 0.352236] Unpacking initramfs... [ 0.729285] Freeing initrd memory: 26012k freed [ 0.748824] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.749611] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.750628] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.751335] Initialise system trusted keyring [ 0.751853] audit: initializing netlink socket (disabled) [ 0.752467] type=2000 audit(1556666801.902:1): initialized [ 0.775314] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.776040] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.778431] zpool: loaded [ 0.778780] zbud: loaded [ 0.779351] VFS: Disk quotas dquot_6.6.0 [ 0.779844] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.780727] msgmni has been set to 3670 [ 0.781260] Key type big_key registered [ 0.781691] SELinux: Registering netfilter hooks [ 0.782563] NET: Registered protocol family 38 [ 0.783104] Key type asymmetric registered [ 0.783560] Asymmetric key parser 'x509' registered [ 0.784144] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 0.784976] io scheduler noop registered [ 0.785417] io scheduler deadline registered (default) [ 0.786028] io scheduler cfq registered [ 0.786459] io scheduler mq-deadline registered [ 0.786981] io scheduler kyber registered [ 0.787544] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.788170] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.788900] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 0.789682] intel_idle: does not run on family 6 model 42 [ 0.789763] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.790548] ACPI: Power Button [PWRF] [ 0.791112] GHES: HEST is not enabled! [ 0.791615] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.813860] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.814990] Non-volatile memory driver v1.3 [ 0.815491] Linux agpgart interface v0.103 [ 0.816087] crash memory driver: version 1.1 [ 0.816604] rdac: device handler registered [ 0.817125] hp_sw: device handler registered [ 0.817597] emc: device handler registered [ 0.818108] alua: device handler registered [ 0.818615] libphy: Fixed MDIO Bus: probed [ 0.819141] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.819853] ehci-pci: EHCI PCI platform driver [ 0.820355] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.821029] ohci-pci: OHCI PCI platform driver [ 0.821522] uhci_hcd: USB Universal Host Controller Interface driver [ 0.823080] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.823711] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.824533] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.825155] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c140 [ 0.825854] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.826571] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.827385] usb usb1: Product: UHCI Host Controller [ 0.827953] usb usb1: Manufacturer: Linux 3.10.0-957.10.1.el7_lustre.x86_64 uhci_hcd [ 0.828775] usb usb1: SerialNumber: 0000:00:01.2 [ 0.829400] hub 1-0:1.0: USB hub found [ 0.829850] hub 1-0:1.0: 2 ports detected [ 0.830469] usbcore: registered new interface driver usbserial_generic [ 0.831211] usbserial: USB Serial support registered for generic [ 0.831906] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.833387] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.833961] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.834648] mousedev: PS/2 mouse device common for all mice [ 0.835588] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.837148] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.838136] rtc_cmos 00:00: RTC can wake from S4 [ 0.838909] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.840129] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.840856] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.841583] cpuidle: using governor menu [ 0.842137] hidraw: raw HID events driver (C) Jiri Kosina [ 0.842859] usbcore: registered new interface driver usbhid [ 0.843479] usbhid: USB HID core driver [ 0.843978] drop_monitor: Initializing network drop monitor service [ 0.844726] TCP: cubic registered [ 0.845145] Initializing XFRM netlink socket [ 0.845721] NET: Registered protocol family 10 [ 0.846448] NET: Registered protocol family 17 [ 0.846975] mpls_gso: MPLS GSO support [ 0.847657] PM: Hibernation image not present or could not be loaded. [ 0.847660] Loading compiled-in X.509 certificates [ 0.848242] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.849291] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.850846] Loaded X.509 cert 'CentOS Linux kernel signing key: 88cbf2ee1fec8cdc233041f7714279b3a857bf26' [ 0.851897] registered taskstats version 1 [ 0.855289] Key type trusted registered [ 0.858330] Key type encrypted registered [ 0.858847] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 0.859772] Magic number: 15:59:455 [ 0.860306] rtc_cmos 00:00: setting system clock to 2019-04-30 23:26:41 UTC (1556666801) [ 0.862090] Freeing unused kernel memory: 1876k freed [ 0.863092] Write protecting the kernel read-only data: 12288k [ 0.864904] Freeing unused kernel memory: 512k freed [ 0.866772] Freeing unused kernel memory: 600k freed [ 0.871214] random: systemd: uninitialized urandom read (16 bytes read) [ 0.872587] random: systemd: uninitialized urandom read (16 bytes read) [ 0.873358] random: systemd: uninitialized urandom read (16 bytes read) [ 0.875041] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.876958] systemd[1]: Detected virtualization kvm. [ 0.877520] systemd[1]: Detected architecture x86-64. [ 0.878090] systemd[1]: Running in initial RAM disk. [ 0.879650] systemd[1]: Set hostname to . [ 0.910993] random: systemd: uninitialized urandom read (16 bytes read) [ 0.911778] random: systemd: uninitialized urandom read (16 bytes read) [ 0.912534] random: systemd: uninitialized urandom read (16 bytes read) [ 0.913319] random: systemd: uninitialized urandom read (16 bytes read) [ 0.914740] random: systemd: uninitialized urandom read (16 bytes read) [ 0.915636] random: systemd: uninitialized urandom read (16 bytes read) [ 0.916792] random: systemd: uninitialized urandom read (16 bytes read) [ 0.922274] systemd[1]: Reached target Swap. [ 0.923731] systemd[1]: Created slice Root Slice. [ 0.925063] systemd[1]: Listening on udev Control Socket. [ 0.926541] systemd[1]: Created slice System Slice. [ 0.927857] systemd[1]: Reached target Slices. [ 0.929134] systemd[1]: Listening on Journal Socket. [ 0.931184] systemd[1]: Starting Apply Kernel Variables... [ 0.932998] systemd[1]: Starting Journal Service... [ 0.934819] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 0.936688] systemd[1]: Reached target Local File Systems. [ 0.938929] systemd[1]: Starting dracut cmdline hook... [ 0.940739] systemd[1]: Starting Setup Virtual Console... [ 0.942797] systemd[1]: Listening on udev Kernel Socket. [ 0.944221] systemd[1]: Reached target Sockets. [ 0.945808] systemd[1]: Reached target Timers. [ 0.951221] systemd[1]: Started Journal Service. [ 1.078350] random: fast init done [ 1.356930] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.358069] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.366160] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffb526003ba000, 52:54:00:a2:92:7f, IRQ 10 [ 1.368613] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.370382] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.371206] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.384947] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.399687] libata version 3.00 loaded. [ 1.404736] ata_piix 0000:00:01.1: version 2.13 [ 1.425946] scsi host0: ata_piix [ 1.433308] scsi host1: ata_piix [ 1.433745] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc180 irq 14 [ 1.434498] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc188 irq 15 [ 1.461087] FDC 0 is a S82078B [ 1.664665] virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X [ 1.664685] virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X [ 1.665681] virtio_blk virtio0: [vda] 251658240 512-byte logical blocks (128 GB/120 GiB) [ 1.676024] vda: vda1 vda2 vda3 vda4 [ 1.748793] tsc: Refined TSC clocksource calibration: 2593.614 MHz [ 6.775125] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ 7.009671] systemd-journald[91]: Received SIGTERM from PID 1 (systemd). [ 7.243736] SELinux: Disabled at runtime. [ 7.244294] SELinux: Unregistering netfilter hooks [ 7.284767] type=1404 audit(1556666807.924:2): selinux=0 auid=4294967295 ses=4294967295 [ 7.300800] ip_tables: (C) 2000-2006 Netfilter Core Team [ 7.301561] systemd[1]: Inserted module 'ip_tables' [ 7.797530] EXT4-fs (vda1): re-mounted. Opts: (null) [ 7.919467] systemd-journald[1270]: Received request to flush runtime journal from PID 1 [ 8.454170] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 8.570822] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 8.621255] cryptd: max_cpu_qlen set to 1000 [ 8.647563] AVX version of gcm_enc/dec engaged. [ 8.648119] AES CTR mode by8 optimization enabled [ 8.712763] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.718547] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 8.907235] ppdev: user-space parallel port driver [ 9.284095] Adding 2098172k swap on /dev/vda2. Priority:-2 extents:1 across:2098172k FS [ 9.349648] type=1305 audit(1556666809.988:3): audit_pid=2458 old=0 auid=4294967295 ses=4294967295 res=1 [ 10.185953] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 10.187465] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 32.043647] random: crng init done [ 298.410185] FS-Cache: Loaded [ 298.435816] RPC: Registered named UNIX socket transport module. [ 298.436489] RPC: Registered udp transport module. [ 298.437024] RPC: Registered tcp transport module. [ 298.437536] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 298.466186] FS-Cache: Netfs 'nfs' registered for caching [ 298.473339] Key type dns_resolver registered [ 298.501099] NFS: Registering the id_resolver key type [ 298.501690] Key type id_resolver registered [ 298.502151] Key type id_legacy registered [ 328.088187] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 328.109806] Loading iSCSI transport class v2.0-870. [ 328.119317] iscsi: registered transport (iser) [ 328.176894] RPC: Registered rdma transport module. [ 328.177480] RPC: Registered rdma backchannel transport module. [ 560.894158] spl: loading out-of-tree module taints kernel. [ 560.897860] spl: module verification failed: signature and/or required key missing - tainting kernel [ 560.904529] SPL: Loaded module v0.7.13-1 [ 560.905209] znvpair: module license 'CDDL' taints kernel. [ 560.905806] Disabling lock debugging due to kernel taint [ 562.983410] ZFS: Loaded module v0.7.13-1, ZFS pool version 5000, ZFS filesystem version 5 [ 691.384294] LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 [ 691.391611] alg: No test for adler32 (adler32-zlib) [ 692.191619] Lustre: Lustre: Build Version: 2.12.1_RC1 [ 692.295714] LNet: Added LNI 10.9.6.155@tcp [8/256/0/180] [ 692.296395] LNet: Accept secure, port 988 [ 693.707829] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 693.709516] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 695.100567] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 10.9.6.154@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.102483] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 695.103520] Lustre: lustre-OST0000: Connection restored to (at 10.9.6.154@tcp) [ 706.769772] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 706.771406] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 712.751432] Lustre: lustre-OST0001: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 712.753366] Lustre: lustre-OST0001: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 716.138634] Lustre: lustre-OST0001: Connection restored to (at 10.9.6.154@tcp) [ 716.353462] Lustre: lustre-OST0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 716.356233] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 745.567542] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 745.824535] Lustre: lustre-OST0000: Recovery over after 0:51, of 2 clients 2 recovered and 0 were evicted. [ 1030.801786] Lustre: DEBUG MARKER: trevis-60vm3.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 1032.151773] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing check_logdir /tmp/test_logs/2019-04-30/234351 [ 1032.874209] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing yml_node [ 1033.371404] Lustre: DEBUG MARKER: Client: Lustre version: 2.10.7 [ 1033.770771] Lustre: DEBUG MARKER: MDS: Lustre version: 2.10.7 [ 1034.168113] Lustre: DEBUG MARKER: OSS: Lustre version: 2.12.1_RC1 [ 1034.422011] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Apr 30 23:43:55 UTC 2019 [ 1034.939592] Lustre: DEBUG MARKER: trevis-60vm3.trevis.whamcloud.com: executing check_config_client /mnt/lustre [ 1037.460986] Lustre: DEBUG MARKER: Using TIMEOUT=100 [ 1044.265011] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1045.236527] Lustre: DEBUG MARKER: excepting tests: 103a 103b 103c 104a 160c 161a 161b 161c 220 225a 225b 228b 407 253 312 42a 42b 42c 65ic 180 [ 1045.468731] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 64b 68 71 115 300o 51b [ 1045.921495] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 23:44:07 (1556667847) [ 1047.070478] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 23:44:08 (1556667848) [ 1048.194178] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 23:44:09 (1556667849) [ 1049.315876] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 23:44:10 (1556667850) [ 1050.581397] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 23:44:11 (1556667851) [ 1051.730418] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 23:44:12 (1556667852) [ 1052.885016] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 23:44:14 (1556667854) [ 1054.042274] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 23:44:15 (1556667855) [ 1055.407273] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 23:44:16 (1556667856) [ 1056.551584] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 23:44:17 (1556667857) [ 1057.698872] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ========= 23:44:18 (1556667858) [ 1058.836288] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 23:44:19 (1556667859) [ 1060.319882] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) ================ 23:44:21 (1556667861) [ 1061.453206] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 23:44:22 (1556667862) [ 1062.684451] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 23:44:23 (1556667863) [ 1063.818414] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 23:44:24 (1556667864) [ 1064.953853] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 23:44:26 (1556667866) [ 1066.087903] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 23:44:27 (1556667867) [ 1067.223829] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 23:44:28 (1556667868) [ 1069.250446] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 23:44:30 (1556667870) [ 1070.388401] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 23:44:31 (1556667871) [ 1071.529539] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 23:44:32 (1556667872) [ 1072.699756] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 23:44:33 (1556667873) [ 1073.847391] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 23:44:35 (1556667875) [ 1074.999101] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 23:44:36 (1556667876) [ 1076.158388] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 23:44:37 (1556667877) [ 1077.306082] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 23:44:38 (1556667878) [ 1078.451302] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 23:44:39 (1556667879) [ 1079.600615] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 23:44:40 (1556667880) [ 1080.783354] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 23:44:41 (1556667881) [ 1081.939596] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 23:44:43 (1556667883) [ 1083.852811] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 23:44:45 (1556667885) [ 1085.248518] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 23:44:46 (1556667886) [ 1086.565424] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 23:44:47 (1556667887) [ 1087.812421] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 23:44:48 (1556667888) [ 1089.017726] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 23:44:50 (1556667890) [ 1089.626950] Lustre: DEBUG MARKER: SKIP: sanity test_17m ldiskfs only test [ 1089.879871] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 23:44:51 (1556667891) [ 1090.491075] Lustre: DEBUG MARKER: SKIP: sanity test_17n ldiskfs only test [ 1090.747144] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 23:44:51 (1556667891) [ 1110.942984] Lustre: lustre-OST0001: Connection restored to (at 10.9.6.154@tcp) [ 1112.799879] Lustre: DEBUG MARKER: trevis-60vm3.trevis.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.960342] Lustre: 4505:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1556667909/real 1556667909] req@ffff8c09db72db00 x1632284173141824/t0(0) o400->MGC10.9.6.154@tcp@10.9.6.154@tcp:26/25 lens 224/224 e 0 to 1 dl 1556667916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1114.960362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.9.6.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1114.965173] Lustre: 4505:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1114.966215] LustreError: 166-1: MGC10.9.6.154@tcp: Connection to MGS (at 10.9.6.154@tcp) was lost; in progress operations using this service will fail [ 1114.967868] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1114.970360] Lustre: Evicted from MGS (at MGC10.9.6.154@tcp_0) after server handle changed from 0xb963bb38e363b00b to 0xb963bb38e363cffb [ 1135.083871] Lustre: lustre-OST0000: deleting orphan objects from 0x0:47 to 0x0:65 [ 1135.083872] Lustre: lustre-OST0001: deleting orphan objects from 0x0:46 to 0x0:65 [ 1136.169960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 23 sec [ 1138.795103] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 23:45:39 (1556667939) [ 1139.918950] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 23:45:41 (1556667941) [ 1141.174606] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 23:45:42 (1556667942) [ 1142.298936] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 23:45:43 (1556667943) [ 1143.426072] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 23:45:44 (1556667944) [ 1144.549169] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 23:45:45 (1556667945) [ 1145.700349] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 23:45:46 (1556667946) [ 1146.864942] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 23:45:48 (1556667948) [ 1148.260374] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 23:45:49 (1556667949) [ 1149.416639] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 23:45:50 (1556667950) [ 1150.558681] Lustre: DEBUG MARKER: == sanity test 23d: file offset is correct after appending writes ==================================== 23:45:51 (1556667951) [ 1151.707836] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 23:45:52 (1556667952) [ 1152.855940] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 23:45:54 (1556667954) [ 1154.010250] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 23:45:55 (1556667955) [ 1155.150132] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 23:45:56 (1556667956) [ 1156.342925] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 23:45:57 (1556667957) [ 1157.490333] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 23:45:58 (1556667958) [ 1158.645055] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 23:45:59 (1556667959) [ 1164.833590] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 23:46:05 (1556667965) [ 1168.985149] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 23:46:10 (1556667970) [ 1170.135453] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 23:46:11 (1556667971) [ 1171.282881] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 23:46:12 (1556667972) [ 1172.440535] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 23:46:13 (1556667973) [ 1173.582820] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 23:46:14 (1556667974) [ 1174.739087] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 23:46:15 (1556667975) [ 1175.908584] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 23:46:17 (1556667977) [ 1225.042697] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 23:47:06 (1556668026) [ 1226.192303] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 23:47:07 (1556668027) [ 1227.353445] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 23:47:08 (1556668028) [ 1228.515500] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 23:47:09 (1556668029) [ 1229.666657] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 23:47:10 (1556668030) [ 1230.821710] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 23:47:11 (1556668031) [ 1232.059455] Lustre: DEBUG MARKER: == sanity test 24v: list large directory (test hash collision, b=17560) ============================== 23:47:13 (1556668033) [ 1239.533733] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 23:47:20 (1556668040) [ 1240.754371] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 23:47:21 (1556668041) [ 1240.988578] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 1241.245816] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 23:47:22 (1556668042) [ 1241.478323] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 1241.738849] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 23:47:22 (1556668042) [ 1260.709536] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 23:47:41 (1556668061) [ 1260.955879] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 1261.226489] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 23:47:42 (1556668062) [ 1261.478684] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 1261.753992] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 23:47:42 (1556668062) [ 1261.988094] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 1262.248525] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 23:47:43 (1556668063) [ 1263.426392] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 23:47:44 (1556668064) [ 1264.574842] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 23:47:45 (1556668065) [ 1265.743802] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 23:47:46 (1556668066) [ 1266.918601] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 23:47:48 (1556668068) [ 1268.108845] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 23:47:49 (1556668069) [ 1269.260765] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 23:47:50 (1556668070) [ 1270.422115] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 23:47:51 (1556668071) [ 1271.649747] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 23:47:52 (1556668072) [ 1272.849605] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 23:47:54 (1556668074) [ 1274.055784] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 23:47:55 (1556668075) [ 1275.250698] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 23:47:56 (1556668076) [ 1276.640801] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 23:47:57 (1556668077) [ 1277.839133] Lustre: DEBUG MARKER: == sanity test 27g: /bin/lfs getstripe with no objects =============================================== 23:47:59 (1556668079) [ 1279.013548] Lustre: DEBUG MARKER: == sanity test 27i: /bin/lfs getstripe with some objects ============================================= 23:48:00 (1556668080) [ 1280.187951] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 23:48:01 (1556668081) [ 1281.358083] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 23:48:02 (1556668082) [ 1282.540796] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 23:48:03 (1556668083) [ 1283.693387] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 1283.946957] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 23:48:05 (1556668085) [ 1300.362654] Lustre: *** cfs_fail_loc=215, val=0*** [ 1318.607091] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 23:48:39 (1556668119) [ 1335.417030] Lustre: *** cfs_fail_loc=215, val=0*** [ 1335.417709] Lustre: Skipped 1 previous similar message [ 1340.424836] Lustre: *** cfs_fail_loc=215, val=1*** [ 1340.425593] Lustre: Skipped 1 previous similar message [ 1345.432714] Lustre: *** cfs_fail_loc=215, val=0*** [ 1345.432715] Lustre: *** cfs_fail_loc=215, val=1*** [ 1365.652629] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 23:49:26 (1556668166) [ 1380.487195] Lustre: *** cfs_fail_loc=215, val=0*** [ 1380.488069] Lustre: Skipped 1 previous similar message [ 1400.242121] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 23:50:01 (1556668201) [ 1415.541638] Lustre: *** cfs_fail_loc=215, val=1*** [ 1415.541639] Lustre: *** cfs_fail_loc=215, val=0*** [ 1447.164996] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 23:50:48 (1556668248) [ 1481.748933] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 23:51:22 (1556668282) [ 1482.924215] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 23:51:24 (1556668284) [ 1484.114857] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 23:51:25 (1556668285) [ 1490.145728] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 23:51:31 (1556668291) [ 1495.666186] Lustre: *** cfs_fail_loc=215, val=1*** [ 1495.666781] Lustre: Skipped 9 previous similar messages [ 1587.225927] Lustre: DEBUG MARKER: == sanity test 27w: check /bin/lfs setstripe -S and getstrip -d options ============================== 23:53:08 (1556668388) [ 1588.418140] Lustre: DEBUG MARKER: == sanity test 27wa: check /bin/lfs setstripe -c -i options ========================================== 23:53:09 (1556668389) [ 1589.633201] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 23:53:10 (1556668390) [ 1601.379073] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 23:53:22 (1556668402) [ 1602.518720] Lustre: DEBUG MARKER: SKIP: sanity test_27y not enough space on OST0 [ 1602.783911] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 23:53:23 (1556668403) [ 1608.520115] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 1608.775886] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x4063 ver 0x0 [ 1609.010438] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x4063, count: 1 [ 1609.256539] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:310/0x136 seq:0 [ 1609.607495] Lustre: Failing over lustre-OST0000 [ 1609.617725] Lustre: server umount lustre-OST0000 complete [ 1610.848867] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.9.6.154@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.260577] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 1617.262580] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1617.770755] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1619.451297] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 1619.452406] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 1619.453503] Lustre: Skipped 4 previous similar messages [ 1626.640542] Lustre: lustre-OST0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 1626.644238] Lustre: lustre-OST0000: deleting orphan objects from 0x0:312 to 0x0:353 [ 1626.942179] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 1627.201026] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x4065 ver 0x0 [ 1627.456114] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x4065, count: 2 [ 1627.719694] Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1107/0x453 seq:0 [ 1628.072994] Lustre: Failing over lustre-OST0001 [ 1628.083014] Lustre: server umount lustre-OST0001 complete [ 1630.880199] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 10.9.6.154@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.816724] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 1634.818756] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 1635.327168] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1637.009670] Lustre: lustre-OST0001: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 1637.010859] Lustre: lustre-OST0001: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 1637.011960] Lustre: Skipped 1 previous similar message [ 1640.795259] Lustre: lustre-OST0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1640.828236] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1108 to 0x0:1153 [ 1641.091871] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:311/0x137 seq:0 [ 1641.449269] Lustre: Failing over lustre-OST0000 [ 1641.458200] Lustre: server umount lustre-OST0000 complete [ 1641.679213] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 10.9.6.154@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.891970] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 1648.893549] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1649.396120] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1651.082907] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 1657.686873] Lustre: lustre-OST0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 1657.691382] Lustre: lustre-OST0000: deleting orphan objects from 0x0:312 to 0x0:385 [ 1658.920780] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 23:54:20 (1556668460) [ 1660.159482] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 23:54:21 (1556668461) [ 1661.364138] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 23:54:22 (1556668462) [ 1662.592500] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 23:54:23 (1556668463) [ 1702.795706] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 23:55:03 (1556668503) [ 1704.013010] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 23:55:05 (1556668505) [ 1704.588348] Lustre: Failing over lustre-OST0000 [ 1704.849646] Lustre: server umount lustre-OST0000 complete [ 1709.529224] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 1709.530918] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1709.562994] Lustre: lustre-OST0000: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 1709.564053] Lustre: lustre-OST0000: Connection restored to (at 10.9.6.154@tcp) [ 1709.564839] Lustre: Skipped 3 previous similar messages [ 1710.029677] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1710.906702] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 10.9.6.154@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1713.063840] Lustre: DEBUG MARKER: trevis-60vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 1716.182009] Lustre: lustre-OST0001: Will be in recovery for at least 2:30, or until 2 clients reconnect [ 1718.650026] Lustre: lustre-OST0000: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. [ 1718.730894] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1169 to 0x0:1185 [ 1719.412534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:397 to 0x0:417 [ 1720.617655] Lustre: DEBUG MARKER: == sanity test 27G: Clear OST pool from stripe ======================================================= 23:55:21 (1556668521) [ 1745.043021] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 23:55:46 (1556668546) [ 1746.328462] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 23:55:47 (1556668547) [ 1747.796321] Lustre: DEBUG MARKER: first d29 [ 1748.047096] Lustre: DEBUG MARKER: second d29 [ 1748.289266] Lustre: DEBUG MARKER: done [ 1749.461050] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 23:55:50 (1556668550) [ 1750.661301] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 23:55:51 (1556668551) [ 1751.854825] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 23:55:53 (1556668553) [ 1753.116258] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 23:55:54 (1556668554) [ 1754.297604] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 23:55:55 (1556668555) [ 1755.496098] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 23:55:56 (1556668556) [ 1756.682902] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 23:55:57 (1556668557) [ 1757.869299] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 23:55:59 (1556668559) [ 1759.066058] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 23:56:00 (1556668560) [ 1766.320692] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 23:56:07 (1556668567) [ 1767.519473] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 23:56:08 (1556668568) [ 1768.721544] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 23:56:09 (1556668569) [ 1769.921733] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 23:56:11 (1556668571) [ 1771.158980] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 23:56:12 (1556668572) [ 1772.385979] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 23:56:13 (1556668573) [ 1773.598013] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 23:56:14 (1556668574) [ 1774.800576] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 23:56:15 (1556668575) [ 1782.970407] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 23:56:24 (1556668584) [ 1783.214212] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 1783.491244] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 23:56:24 (1556668584) [ 1785.049368] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 23:56:26 (1556668586) [ 1786.373839] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 23:56:27 (1556668587) [ 1787.744935] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 23:56:28 (1556668588) [ 1789.135863] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 23:56:30 (1556668590) [ 1790.327435] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 23:56:31 (1556668591) [ 1791.543616] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 23:56:32 (1556668592) [ 1792.754545] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 23:56:33 (1556668593) [ 1793.969035] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 23:56:35 (1556668595) [ 1795.315358] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 23:56:36 (1556668596) [ 1796.580317] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 23:56:37 (1556668597) [ 1797.899279] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 23:56:39 (1556668599) [ 1799.257437] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 23:56:40 (1556668600) [ 1800.455408] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 23:56:41 (1556668601) [ 1801.657609] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 23:56:42 (1556668602) [ 1802.864979] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 23:56:44 (1556668604) [ 1803.111594] Lustre: DEBUG MARKER: 32p_1 [ 1803.361031] Lustre: DEBUG MARKER: 32p_2 [ 1803.612558] Lustre: DEBUG MARKER: 32p_3 [ 1803.864927] Lustre: DEBUG MARKER: 32p_4 [ 1804.122574] Lustre: DEBUG MARKER: 32p_5 [ 1804.366403] Lustre: DEBUG MARKER: 32p_6 [ 1804.616832] Lustre: DEBUG MARKER: 32p_7 [ 1804.864483] Lustre: DEBUG MARKER: 32p_8 [ 1805.118920] Lustre: DEBUG MARKER: 32p_9 [ 1805.369530] Lustre: DEBUG MARKER: 32p_10 [ 1806.554102] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 23:56:47 (1556668607) [ 1807.986244] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 23:56:49 (1556668609) [ 1809.353794] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 23:56:50 (1556668610) [ 1809.618324] Lustre: DEBUG MARKER: 33_1 [ 1809.867589] Lustre: DEBUG MARKER: 33_2 [ 1811.046857] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 23:56:52 (1556668612) [ 1812.820857] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 23:56:54 (1556668614) [ 1814.629630] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 23:56:55 (1556668615) [ 1816.217501] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 23:56:57 (1556668617) [ 1816.458263] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 1816.725213] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 23:56:57 (1556668617) [ 1816.985251] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 1817.259266] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 23:56:58 (1556668618) [ 1817.501706] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 1817.769207] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 23:56:58 (1556668618) [ 1818.957051] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 23:57:00 (1556668620) [ 1820.153179] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 23:57:01 (1556668621) [ 1821.345097] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 23:57:02 (1556668622) [ 1822.532876] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 23:57:03 (1556668623) [ 1823.726683] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 23:57:04 (1556668624) [ 1824.925605] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 23:57:06 (1556668626) [ 1826.136727] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 23:57:07 (1556668627) [ 1827.379220] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 23:57:08 (1556668628) [ 1832.436272] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 23:57:13 (1556668633) [ 1833.697339] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 23:57:14 (1556668634) [ 1834.952795] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 23:57:16 (1556668636) [ 1836.138912] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 23:57:17 (1556668637) [ 1837.701776] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 23:57:18 (1556668638) [ 1838.900642] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 23:57:20 (1556668640) [ 1840.097957] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 23:57:21 (1556668641) [ 1842.334590] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 23:57:23 (1556668643) [ 1966.068742] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 23:59:27 (1556668767) [ 1968.296618] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 23:59:29 (1556668769) [ 1968.540429] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 1968.809657] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 23:59:30 (1556668770) [ 1969.997089] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ========================================================== 23:59:31 (1556668771) [ 1973.209133] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 23:59:34 (1556668774) [ 1975.557440] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 23:59:36 (1556668776) [ 1978.795043] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 23:59:39 (1556668779) [ 1980.016913] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 23:59:41 (1556668781) [ 1981.230405] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 23:59:42 (1556668782) [ 1984.455192] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 23:59:45 (1556668785) [ 1987.681850] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 23:59:48 (1556668788) [ 1989.911991] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 23:59:51 (1556668791) [ 1992.142702] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 23:59:53 (1556668793) [ 1996.616315] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 23:59:57 (1556668797) [ 1998.942861] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 00:00:00 (1556668800) [ 2006.902660] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 00:00:08 (1556668808) [ 2010.130217] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 00:00:11 (1556668811) [ 2021.904687] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 00:00:23 (1556668823) [ 2023.123731] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 00:00:24 (1556668824) [ 2023.368930] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 2023.640590] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 00:00:24 (1556668824) [ 2024.854725] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 00:00:26 (1556668826) [ 2026.085210] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 00:00:27 (1556668827) [ 2028.315821] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 2028.569855] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 2028.824941] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 2029.100243] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ============================ 00:00:30 (1556668830) [ 2031.153980] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 00:00:32 (1556668832) [ 2051.100979] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 00:00:52 (1556668852) [ 2053.373474] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 00:00:54 (1556668854) [ 2054.615152] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 00:00:55 (1556668855) [ 2055.841673] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre ======================================================= 00:00:57 (1556668857) [ 2057.151005] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ============================================ 00:00:58 (1556668858) [ 2058.375171] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 00:00:59 (1556668859) [ 2059.931761] Lustre: DEBUG MARKER: == sanity test 45: osc io page accounting ============================================================ 00:01:01 (1556668861) [ 2061.710645] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 00:01:02 (1556668862) [ 2063.390906] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 00:01:04 (1556668864) [ 2064.846528] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 00:01:06 (1556668866) [ 2066.100969] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 00:01:07 (1556668867) [ 2067.341616] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 00:01:08 (1556668868) [ 2068.580413] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 00:01:09 (1556668869) [ 2069.818178] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 00:01:11 (1556668871) [ 2071.249500] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 00:01:12 (1556668872) [ 2072.468991] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 00:01:13 (1556668873) [ 2073.957719] Lustre: DEBUG MARKER: SKIP: sanity test_51b skipping SLOW test 51b [ 2074.221294] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 00:01:15 (1556668875) [ 2074.471256] Lustre: DEBUG MARKER: SKIP: sanity test_51d needs >= 3 OSTs [ 2074.741815] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 00:01:15 (1556668875) [ 2074.995585] Lustre: DEBUG MARKER: SKIP: sanity test_51e ldiskfs only test [ 2075.266516] Lustre: DEBUG MARKER: == sanity test 51f: check many open files limit ====================================================== 00:01:16 (1556668876) [ 2452.470943] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 00:07:33 (1556669253) [ 2453.819965] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 00:07:35 (1556669255) [ 2455.106045] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 00:07:36 (1556669256) [ 2456.304781] Lustre: DEBUG MARKER: sanity test_53: @@@@@@ FAIL: can not match last_seq/last_id for *OST*-osc- [ 2469.420849] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 00:07:50 (1556669270) [ 2470.831805] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 00:07:52 (1556669272) [ 2472.041430] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 00:07:53 (1556669273) [ 2474.648449] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 00:07:55 (1556669275) [ 2475.848352] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 00:07:57 (1556669277) [ 2477.055740] Lustre: DEBUG MARKER: == sanity test 56a: check /bin/lfs getstripe ========================================================= 00:07:58 (1556669278) [ 2478.461076] Lustre: DEBUG MARKER: == sanity test 56b: check /bin/lfs getdirstripe ====================================================== 00:07:59 (1556669279) [ 2479.705767] Lustre: DEBUG MARKER: == sanity test 56c: check 'lfs df' showing device status ============================================= 00:08:00 (1556669280) [ 2501.643587] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 00:08:22 (1556669302) [ 2502.973947] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 00:08:24 (1556669304) [ 2504.314500] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 00:08:25 (1556669305) [ 2505.541518] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 00:08:26 (1556669306) [ 2506.907092] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 00:08:28 (1556669308) [ 2508.273843] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 00:08:29 (1556669309) [ 2509.637944] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 00:08:30 (1556669310) [ 2511.016653] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 00:08:32 (1556669312) [ 2517.383413] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 00:08:38 (1556669318) [ 2520.065346] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 00:08:41 (1556669321) [ 2521.431388] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 00:08:42 (1556669322) [ 2522.780993] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 00:08:43 (1556669323) [ 2524.460968] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 00:08:45 (1556669325) [ 2526.010031] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 00:08:47 (1556669327) [ 2527.595665] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 00:08:48 (1556669328) [ 2529.087057] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 00:08:50 (1556669330) [ 2530.537166] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 00:08:51 (1556669331) [ 2548.363990] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 00:09:09 (1556669349) [ 2549.989555] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 00:09:11 (1556669351) [ 2551.949095] Lustre: DEBUG MARKER: == sanity test 56xb: lfs migration hard link support ================================================= 00:09:13 (1556669353) [ 2576.537308] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 00:09:37 (1556669377) [ 2577.984432] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 00:09:39 (1556669379) [ 2579.551902] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 00:09:40 (1556669380) [ 2579.802541] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 2580.079516] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 00:09:41 (1556669381) [ 2580.334868] Lustre: DEBUG MARKER: SKIP: sanity test_57a ldiskfs only test [ 2580.609934] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 00:09:41 (1556669381) [ 2580.870465] Lustre: DEBUG MARKER: SKIP: sanity test_57b ldiskfs only test [ 2581.145695] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 00:09:42 (1556669382) [ 2582.436219] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 00:09:43 (1556669383) [ 2598.147137] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 00:09:59 (1556669399) [ 2598.570019] Lustre: DEBUG MARKER: test_60 run 25023 - from kernel mode [ 2664.772663] Lustre: lustre-OST0001: Connection restored to (at 10.9.6.154@tcp) [ 2664.773910] Lustre: Skipped 3 previous similar messages [ 2665.610114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 10.9.6.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2665.611919] Lustre: Skipped 1 previous similar message [ 2665.612522] LustreError: 166-1: MGC10.9.6.154@tcp: Connection to MGS (at 10.9.6.154@tcp) was lost; in progress operations using this service will fail [ 2665.613123] LustreError: 167-0: lustre-MDT0000-lwp-OST0000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2665.613124] LustreError: Skipped 1 previous similar message [ 2665.616915] Lustre: Evicted from MGS (at 10.9.6.154@tcp) after server handle changed from 0xb963bb38e38013a1 to 0xb963bb38e4083361 [ 2666.797388] Lustre: lustre-OST0000: deleting orphan objects from 0x0:51627 to 0x0:51649 [ 2666.797690] Lustre: lustre-OST0001: deleting orphan objects from 0x0:52384 to 0x0:52417 [ 2667.246336] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 00:11:08 (1556669468) [ 2668.471886] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 00:11:09 (1556669469) [ 2690.291733] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 00:11:31 (1556669491) [ 2691.523847] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 00:11:32 (1556669492) [ 2693.027560] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 00:11:34 (1556669494) [ 2694.708594] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 00:11:35 (1556669495) [ 2756.656654] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 00:12:37 (1556669557) [ 2761.753611] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 00:12:42 (1556669562) [ 2763.420330] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 2763.702130] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ =============================== 00:12:44 (1556669564) [ 2765.416622] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ========================================================= 00:12:46 (1556669566) [ 2812.212675] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 00:13:33 (1556669613) [ 2813.508563] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 00:13:34 (1556669614) [ 2814.743523] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 =================================== 00:13:35 (1556669615) [ 2815.976848] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 00:13:37 (1556669617) [ 2817.231049] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 00:13:38 (1556669618) [ 2818.462634] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 00:13:39 (1556669619) [ 2819.693579] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 00:13:40 (1556669620) [ 2820.930798] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 00:13:42 (1556669622) [ 2822.176667] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ===================================== 00:13:43 (1556669623) [ 2826.832477] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 00:13:48 (1556669628) [ 2829.293858] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 00:13:50 (1556669630) [ 2829.842881] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 10.9.6.154@tcp) reconnecting [ 2829.844025] Lustre: lustre-OST0001: Connection restored to (at 10.9.6.154@tcp) [ 2829.844805] Lustre: Skipped 3 previous similar messages [ 2829.846953] Lustre: lustre-OST0001: deleting orphan objects from 0x0:54926 to 0x0:54945 [ 2830.015851] Lustre: lustre-OST0000: deleting orphan objects from 0x0:54155 to 0x0:54177 [ 2835.240890] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 10.9.6.154@tcp) reconnecting [ 2835.241929] Lustre: Skipped 1 previous similar message [ 2835.244893] Lustre: lustre-OST0001: deleting orphan objects from 0x0:55946 to 0x0:55969 [ 2841.580165] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 10.9.6.154@tcp) reconnecting [ 2841.583410] Lustre: lustre-OST0000: deleting orphan objects from 0x0:55180 to 0x0:55201 [ 2843.983598] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 00:14:05 (1556669645) [ 2845.274817] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 00:14:06 (1556669646) [ 2846.509378] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ==================== 00:14:07 (1556669647) [ 2888.410843] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 00:14:49 (1556669689) [ 2897.560137] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 00:14:58 (1556669698) [ 2898.896792] Lustre: *** cfs_fail_loc=217, val=0*** [ 2901.161941] Lustre: *** cfs_fail_loc=217, val=0*** [ 2901.162547] Lustre: Skipped 7 previous similar messages [ 2902.580824] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 2902.849787] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 00:15:04 (1556669704) [ 2904.490541] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 00:15:05 (1556669705) [ 2907.006216] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 00:15:08 (1556669708) [ 2934.273421] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 00:15:35 (1556669735) [ 2935.501042] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 00:15:36 (1556669736) [ 2936.727343] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 00:15:37 (1556669737) [ 2937.934630] Lustre: DEBUG MARKER: == sanity test 76: confirm clients recycle inodes properly =========================================== 00:15:39 (1556669739) [ 2940.010426] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 00:15:41 (1556669741) [ 2941.591622] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 00:15:42 (1556669742) [ 2941.665645] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eec9:0x0] object 0x0:56041 extent [0-1048575]: client csum 2b489527, server csum 2b489526 [ 2943.146658] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2944.302340] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.9.6.156@tcp inode [0x200000401:0x1eec9:0x0] object 0x0:56041 extent [0-1048575], client returned csum a386d26 (type 1), server csum f22a3795 (type 1) [ 2944.828255] Lustre: DEBUG MARKER: set checksum type to adler [ 2945.932373] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.9.6.156@tcp inode [0x200000401:0x1eec9:0x0] object 0x0:56041 extent [0-1048575], client returned csum 484cea05 (type 2), server csum e60eebc2 (type 2) [ 2946.462531] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2947.538814] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.9.6.156@tcp inode [0x200000401:0x1eec9:0x0] object 0x0:56041 extent [0-1048575], client returned csum 797fb817 (type 4), server csum 2b489526 (type 4) [ 2948.014775] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2949.270454] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================= 00:15:50 (1556669750) [ 2951.798066] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.9.6.156@tcp inode [0x200000401:0x1eeca:0x0] object 0x0:55250 extent [0-1048575], client returned csum 797fb817 (type 4), server csum 2b489526 (type 4) [ 2967.500255] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 00:16:08 (1556669768) [ 2967.606757] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eecc:0x0] object 0x0:55251 extent [0-1048575]: client csum b5ea7f3c, server csum b5ea7f3b [ 2970.832718] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.9.6.156@tcp inode [0x200000401:0x1eecc:0x0] object 0x0:55251 extent [0-1048575], client returned csum fe0f685f (type 4), server csum b5ea7f3b (type 4) [ 2972.132262] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 00:16:13 (1556669773) [ 2972.405993] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2972.495499] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11 [ 2976.398968] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [7340032-8388607]: client csum b2f1b12, server csum b2f1b11 [ 2976.403405] LustreError: Skipped 15 previous similar messages [ 2980.398765] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [6291456-7340031]: client csum b2f1b12, server csum b2f1b11 [ 2980.403952] LustreError: Skipped 7 previous similar messages [ 2989.684202] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [6291456-7340031]: client csum b2f1b12, server csum b2f1b11 [ 2989.686246] LustreError: Skipped 15 previous similar messages [ 3006.348623] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [2097152-3145727]: client csum b2f1b12, server csum b2f1b11 [ 3006.353013] LustreError: Skipped 19 previous similar messages [ 3035.066644] Lustre: DEBUG MARKER: set checksum type to adler [ 3038.599857] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [0-1048575]: client csum 19eeae62, server csum 19eeae61 [ 3038.601794] LustreError: Skipped 43 previous similar messages [ 3097.320134] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3104.646994] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-10.9.6.156@tcp inode [0x200000401:0x1eece:0x0] object 0x0:56042 extent [7340032-8388607]: client csum b5ea7f3c, server csum b5ea7f3b [ 3104.648892] LustreError: Skipped 95 previous similar messages [ 3160.322954] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3161.559193] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 00:19:22 (1556669962) [ 3161.816986] Lustre: *** cfs_fail_loc=21a, val=0*** [ 3163.314695] Lustre: *** cfs_fail_loc=21b, val=0*** [ 3165.954010] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 00:19:27 (1556669967) [ 3166.604561] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 3166.604562] Lustre: lustre-OST0001: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 3166.604565] Lustre: Skipped 3 previous similar messages [ 3166.994259] Lustre: DEBUG MARKER: trevis-60vm3.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff95a9287ab000.ost_server_uuid 200 [ 3167.373056] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95a9287ab000.ost_server_uuid in FULL state after 0 sec [ 3167.745606] Lustre: DEBUG MARKER: trevis-60vm3.trevis.whamcloud.com: executing wait_import_state FULL osc.lustre-OST0001-osc-ffff95a9287ab000.ost_server_uuid 200 [ 3167.982512] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95a9287ab000.ost_server_uuid in FULL state after 0 sec [ 3169.515293] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 00:19:30 (1556669970) [ 3180.362967] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 00:19:41 (1556669981) [ 3195.506046] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 00:19:56 (1556669996) [ 3203.560651] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 00:20:04 (1556670004) [ 3203.785639] Lustre: *** cfs_fail_loc=228, val=0*** [ 3205.522363] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 00:20:06 (1556670006) [ 3205.737059] Lustre: *** cfs_fail_loc=228, val=0*** [ 3208.068617] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 00:20:09 (1556670009) [ 3214.527077] Lustre: DEBUG MARKER: == sanity test 83: Short write in ptask ============================================================== 00:20:15 (1556670015) [ 3216.086136] Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 00:20:17 (1556670017) [ 3218.176163] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 00:20:19 (1556670019) [ 3219.516016] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 00:20:20 (1556670020) [ 3221.741996] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 00:20:22 (1556670022) [ 3228.031914] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 00:20:29 (1556670029) [ 3229.322309] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 00:20:30 (1556670030) [ 3231.949360] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 00:20:33 (1556670033) [ 3233.617439] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 00:20:34 (1556670034) [ 3255.193205] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 00:20:56 (1556670056) [ 3259.011883] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 00:21:00 (1556670060) [ 3268.291244] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 00:21:09 (1556670069) [ 3280.980154] WARNING: MMP writes to pool 'lustre-ost1' have not succeeded in over 5s; suspending pool [ 3280.981448] WARNING: Pool 'lustre-ost1' has encountered an uncorrectable I/O failure and has been suspended. [ 3281.091886] WARNING: MMP writes to pool 'lustre-ost2' have not succeeded in over 5s; suspending pool [ 3281.092868] WARNING: Pool 'lustre-ost2' has encountered an uncorrectable I/O failure and has been suspended. [ 3474.405076] LNet: Service thread pid 30189 was inactive for 200.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3474.409289] Pid: 30189, comm: ll_ost_io00_000 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 3474.410361] Call Trace: [ 3474.410694] [] cv_wait_common+0x125/0x150 [spl] [ 3474.411403] [] __cv_wait+0x15/0x20 [spl] [ 3474.412004] [] txg_wait_synced+0xef/0x140 [zfs] [ 3474.412817] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 3474.413488] [] dmu_tx_assign+0x92/0x490 [zfs] [ 3474.414163] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 3474.414896] [] ofd_trans_start+0x75/0xf0 [ofd] [ 3474.415596] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 3474.416312] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 3474.416962] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 3474.417923] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 3474.418699] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3474.419550] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 3474.420261] [] kthread+0xd1/0xe0 [ 3474.420817] [] ret_from_fork_nospec_end+0x0/0x39 [ 3474.421507] [] 0xffffffffffffffff [ 3474.422188] LustreError: dumping log to /tmp/lustre-log.1556670275.30189 [ 3475.750504] LNet: Service thread pid 12764 was inactive for 201.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3475.752248] Pid: 12764, comm: ll_ost_io00_008 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 3475.753284] Call Trace: [ 3475.753560] [] cv_wait_common+0x125/0x150 [spl] [ 3475.754289] [] __cv_wait+0x15/0x20 [spl] [ 3475.754893] [] txg_wait_synced+0xef/0x140 [zfs] [ 3475.755595] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 3475.756352] [] dmu_tx_assign+0x92/0x490 [zfs] [ 3475.757038] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 3475.757752] [] ofd_trans_start+0x75/0xf0 [ofd] [ 3475.758398] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 3475.759149] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 3475.759808] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 3475.760607] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 3475.761356] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3475.762208] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 3475.762958] [] kthread+0xd1/0xe0 [ 3475.763544] [] ret_from_fork_nospec_end+0x0/0x39 [ 3475.764201] [] 0xffffffffffffffff [ 3475.764772] Pid: 12857, comm: ll_ost_io00_010 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 3475.765830] Call Trace: [ 3475.766124] [] cv_wait_common+0x125/0x150 [spl] [ 3475.766806] [] __cv_wait+0x15/0x20 [spl] [ 3475.767396] [] txg_wait_synced+0xef/0x140 [zfs] [ 3475.768132] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 3475.768798] [] dmu_tx_assign+0x92/0x490 [zfs] [ 3475.769549] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 3475.770241] [] ofd_trans_start+0x75/0xf0 [ofd] [ 3475.770901] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 3475.771660] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 3475.772294] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 3475.773047] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 3475.773817] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3475.774697] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 3475.775382] [] kthread+0xd1/0xe0 [ 3475.775920] [] ret_from_fork_nospec_end+0x0/0x39 [ 3475.776670] [] 0xffffffffffffffff [ 3475.777209] Pid: 12762, comm: ll_ost_io00_006 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 3475.778294] Call Trace: [ 3475.778588] [] cv_wait_common+0x125/0x150 [spl] [ 3475.779236] [] __cv_wait+0x15/0x20 [spl] [ 3475.779862] [] txg_wait_synced+0xef/0x140 [zfs] [ 3475.780598] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 3475.781247] [] dmu_tx_assign+0x92/0x490 [zfs] [ 3475.781909] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 3475.782708] [] ofd_trans_start+0x75/0xf0 [ofd] [ 3475.783353] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 3475.784073] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 3475.784736] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 3475.785453] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 3475.786225] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3475.787084] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 3475.787794] [] kthread+0xd1/0xe0 [ 3475.788317] [] ret_from_fork_nospec_end+0x0/0x39 [ 3475.789029] [] 0xffffffffffffffff [ 3475.789601] Pid: 12765, comm: ll_ost_io00_009 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 3475.790641] Call Trace: [ 3475.790904] [] cv_wait_common+0x125/0x150 [spl] [ 3475.791702] [] __cv_wait+0x15/0x20 [spl] [ 3475.792293] [] txg_wait_synced+0xef/0x140 [zfs] [ 3475.793000] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 3475.793671] [] dmu_tx_assign+0x92/0x490 [zfs] [ 3475.794322] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 3475.795071] [] ofd_trans_start+0x75/0xf0 [ofd] [ 3475.795737] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 3475.796429] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 3475.797174] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 3475.797915] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 3475.798720] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 3475.799548] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 3475.800262] [] kthread+0xd1/0xe0 [ 3475.800803] [] ret_from_fork_nospec_end+0x0/0x39 [ 3475.801585] [] 0xffffffffffffffff [ 3475.802141] LNet: Service thread pid 30191 was inactive for 201.84s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3868.515246] Lustre: 20133:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8c0a3b251850 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:350/0 lens 608/448 e 23 to 0 dl 1556670675 ref 2 fl Interpret:/0/0 rc 0/0 [ 3875.155615] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 3875.157060] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 3875.158171] Lustre: Skipped 2 previous similar messages [ 4471.734971] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c0a37a69850 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:196/0 lens 608/0 e 0 to 0 dl 1556671276 ref 2 fl New:/2/ffffffff rc 0/-1 [ 4471.737946] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 4476.137874] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 4476.139131] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 5071.770066] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c0a37a6f850 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:42/0 lens 608/0 e 0 to 0 dl 1556671877 ref 2 fl New:/2/ffffffff rc 0/-1 [ 5071.775314] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 5075.865723] LNet: Service thread pid 12763 was inactive for 1200.76s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 5075.867516] LNet: Skipped 3 previous similar messages [ 5075.868050] Pid: 12763, comm: ll_ost_io00_007 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5075.869120] Call Trace: [ 5075.869440] [] cv_wait_common+0x125/0x150 [spl] [ 5075.870138] [] __cv_wait+0x15/0x20 [spl] [ 5075.870745] [] txg_wait_synced+0xef/0x140 [zfs] [ 5075.871526] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5075.872203] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5075.872869] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5075.873595] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5075.874290] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5075.875004] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5075.875658] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5075.876590] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5075.877370] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5075.878225] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5075.878935] [] kthread+0xd1/0xe0 [ 5075.879490] [] ret_from_fork_nospec_end+0x0/0x39 [ 5075.880185] [] 0xffffffffffffffff [ 5075.880846] LustreError: dumping log to /tmp/lustre-log.1556671877.12763 [ 5076.047081] Pid: 7608, comm: ll_ost_io00_003 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5076.048200] Call Trace: [ 5076.048477] [] cv_wait_common+0x125/0x150 [spl] [ 5076.049155] [] __cv_wait+0x15/0x20 [spl] [ 5076.049762] [] txg_wait_synced+0xef/0x140 [zfs] [ 5076.050451] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5076.051118] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5076.051785] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5076.052493] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5076.053157] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5076.053866] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5076.054507] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5076.055252] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5076.056016] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5076.056865] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5076.057560] [] kthread+0xd1/0xe0 [ 5076.058103] [] ret_from_fork_nospec_end+0x0/0x39 [ 5076.058774] [] 0xffffffffffffffff [ 5076.059326] Pid: 20132, comm: ll_ost_io00_016 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5076.060396] Call Trace: [ 5076.060674] [] cv_wait_common+0x125/0x150 [spl] [ 5076.061340] [] __cv_wait+0x15/0x20 [spl] [ 5076.061943] [] txg_wait_synced+0xef/0x140 [zfs] [ 5076.062624] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5076.063300] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5076.063965] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5076.064680] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5076.065331] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5076.066035] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5076.066686] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5076.067415] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5076.068186] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5076.069023] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5076.069739] [] kthread+0xd1/0xe0 [ 5076.070274] [] ret_from_fork_nospec_end+0x0/0x39 [ 5076.070946] [] 0xffffffffffffffff [ 5076.071505] Pid: 12713, comm: ll_ost_io00_004 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5076.072576] Call Trace: [ 5076.072858] [] cv_wait_common+0x125/0x150 [spl] [ 5076.073523] [] __cv_wait+0x15/0x20 [spl] [ 5076.074137] [] txg_wait_synced+0xef/0x140 [zfs] [ 5076.074828] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5076.075487] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5076.076164] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5076.076879] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5076.077539] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5076.078263] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5076.078916] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5076.079649] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5076.080428] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5076.081279] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5076.081991] [] kthread+0xd1/0xe0 [ 5076.082529] [] ret_from_fork_nospec_end+0x0/0x39 [ 5076.083211] [] 0xffffffffffffffff [ 5076.083760] Pid: 20131, comm: ll_ost_io00_015 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5076.084833] Call Trace: [ 5076.085108] [] cv_wait_common+0x125/0x150 [spl] [ 5076.085775] [] __cv_wait+0x15/0x20 [spl] [ 5076.086381] [] txg_wait_synced+0xef/0x140 [zfs] [ 5076.087076] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5076.087749] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5076.088420] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5076.089156] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5076.089822] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5076.090532] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5076.091192] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5076.091930] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5076.092705] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5076.093548] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5076.094267] [] kthread+0xd1/0xe0 [ 5076.094802] [] ret_from_fork_nospec_end+0x0/0x39 [ 5076.095470] [] 0xffffffffffffffff [ 5076.096069] LNet: Service thread pid 20134 was inactive for 1200.99s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 5076.097415] LNet: Skipped 3 previous similar messages [ 5077.117668] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 5077.118843] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 5077.127115] Lustre: 3664:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c0a37a6b050 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:42/0 lens 608/0 e 0 to 0 dl 1556671877 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 5673.852657] LNet: Service thread pid 22690 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 5673.854527] LNet: Skipped 4 previous similar messages [ 5673.855072] Pid: 22690, comm: ll_ost_io00_019 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5673.856310] Call Trace: [ 5673.856677] [] cv_wait_common+0x125/0x150 [spl] [ 5673.857370] [] __cv_wait+0x15/0x20 [spl] [ 5673.857989] [] txg_wait_synced+0xef/0x140 [zfs] [ 5673.858794] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5673.859491] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5673.860232] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5673.860975] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5673.861707] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5673.862504] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5673.863233] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5673.864233] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5673.865023] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5673.865908] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5673.866654] [] kthread+0xd1/0xe0 [ 5673.867212] [] ret_from_fork_nospec_end+0x0/0x39 [ 5673.867903] [] 0xffffffffffffffff [ 5673.868582] LustreError: dumping log to /tmp/lustre-log.1556672475.22690 [ 5673.870902] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c0a3b252850 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:643/0 lens 608/0 e 0 to 0 dl 1556672478 ref 2 fl New:/2/ffffffff rc 0/-1 [ 5673.876900] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 5677.948375] LNet: Service thread pid 23590 was inactive for 1201.86s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 5677.950192] Pid: 23590, comm: ll_ost_io00_020 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 5677.951267] Call Trace: [ 5677.951558] [] cv_wait_common+0x125/0x150 [spl] [ 5677.952347] [] __cv_wait+0x15/0x20 [spl] [ 5677.953006] [] txg_wait_synced+0xef/0x140 [zfs] [ 5677.953710] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 5677.954386] [] dmu_tx_assign+0x92/0x490 [zfs] [ 5677.955056] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 5677.955769] [] ofd_trans_start+0x75/0xf0 [ofd] [ 5677.956445] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 5677.957158] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 5677.957817] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 5677.958571] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 5677.959340] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 5677.960208] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 5677.960923] [] kthread+0xd1/0xe0 [ 5677.961460] [] ret_from_fork_nospec_end+0x0/0x39 [ 5677.962131] [] 0xffffffffffffffff [ 5677.962687] LustreError: dumping log to /tmp/lustre-log.1556672479.23590 [ 5678.094072] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 5678.095232] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 5678.099102] Lustre: 3720:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c0a37a6d850 x1632281923889024/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:643/0 lens 608/0 e 0 to 0 dl 1556672478 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 5678.102027] Lustre: 3720:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 6271.838417] LNet: Service thread pid 3662 was inactive for 1200.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 6271.840277] Pid: 3662, comm: ll_ost_io00_021 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 6271.841355] Call Trace: [ 6271.841691] [] cv_wait_common+0x125/0x150 [spl] [ 6271.842404] [] __cv_wait+0x15/0x20 [spl] [ 6271.843094] [] txg_wait_synced+0xef/0x140 [zfs] [ 6271.844055] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 6271.844743] [] dmu_tx_assign+0x92/0x490 [zfs] [ 6271.845462] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 6271.846225] [] ofd_trans_start+0x75/0xf0 [ofd] [ 6271.846995] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 6271.847723] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 6271.848421] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 6271.849396] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 6271.850186] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 6271.851102] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 6271.851817] [] kthread+0xd1/0xe0 [ 6271.852398] [] ret_from_fork_nospec_end+0x0/0x39 [ 6271.853122] [] 0xffffffffffffffff [ 6271.853875] LustreError: dumping log to /tmp/lustre-log.1556673073.3662 [ 6273.886415] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09f3af3050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:489/0 lens 608/0 e 0 to 0 dl 1556673079 ref 2 fl New:/2/ffffffff rc 0/-1 [ 6273.889606] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 6279.071694] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 6279.073130] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 6279.077273] Lustre: 3760:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c0a3b252050 x1632281923888736/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:489/0 lens 608/0 e 0 to 0 dl 1556673079 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 6279.080190] Lustre: 3760:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 6280.029982] LNet: Service thread pid 3664 was inactive for 1202.95s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 6280.032028] Pid: 3664, comm: ll_ost_io00_022 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 6280.033124] Call Trace: [ 6280.033425] [] cv_wait_common+0x125/0x150 [spl] [ 6280.034220] [] __cv_wait+0x15/0x20 [spl] [ 6280.034905] [] txg_wait_synced+0xef/0x140 [zfs] [ 6280.035685] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 6280.036501] [] dmu_tx_assign+0x92/0x490 [zfs] [ 6280.037206] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 6280.038014] [] ofd_trans_start+0x75/0xf0 [ofd] [ 6280.038769] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 6280.039590] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 6280.040280] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 6280.041126] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 6280.041944] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 6280.042810] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 6280.043549] [] kthread+0xd1/0xe0 [ 6280.044115] [] ret_from_fork_nospec_end+0x0/0x39 [ 6280.044808] [] 0xffffffffffffffff [ 6280.045436] LustreError: dumping log to /tmp/lustre-log.1556673081.3664 [ 6875.967619] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09f3803050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:335/0 lens 608/0 e 0 to 0 dl 1556673680 ref 2 fl New:/2/ffffffff rc 0/-1 [ 6875.970606] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 6878.015149] LNet: Service thread pid 3717 was inactive for 1204.20s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 6878.016963] Pid: 3717, comm: ll_ost_io00_023 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 6878.018035] Call Trace: [ 6878.018381] [] cv_wait_common+0x125/0x150 [spl] [ 6878.019076] [] __cv_wait+0x15/0x20 [spl] [ 6878.019720] [] txg_wait_synced+0xef/0x140 [zfs] [ 6878.020537] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 6878.021245] [] dmu_tx_assign+0x92/0x490 [zfs] [ 6878.021909] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 6878.022664] [] ofd_trans_start+0x75/0xf0 [ofd] [ 6878.023368] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 6878.024090] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 6878.024762] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 6878.025708] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 6878.026498] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 6878.027369] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 6878.028075] [] kthread+0xd1/0xe0 [ 6878.028656] [] ret_from_fork_nospec_end+0x0/0x39 [ 6878.029362] [] 0xffffffffffffffff [ 6878.030017] LustreError: dumping log to /tmp/lustre-log.1556673679.3717 [ 6880.051140] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 6880.052373] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 6880.055983] Lustre: 3806:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c0a3b250050 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:335/0 lens 608/0 e 0 to 0 dl 1556673680 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 6880.058889] Lustre: 3806:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 6882.110923] Pid: 3720, comm: ll_ost_io00_024 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 6882.112073] Call Trace: [ 6882.112349] [] cv_wait_common+0x125/0x150 [spl] [ 6882.113037] [] __cv_wait+0x15/0x20 [spl] [ 6882.113637] [] txg_wait_synced+0xef/0x140 [zfs] [ 6882.114343] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 6882.115012] [] dmu_tx_assign+0x92/0x490 [zfs] [ 6882.115664] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 6882.116385] [] ofd_trans_start+0x75/0xf0 [ofd] [ 6882.117053] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 6882.117753] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 6882.118410] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 6882.119169] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 6882.119944] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 6882.120782] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 6882.121498] [] kthread+0xd1/0xe0 [ 6882.122042] [] ret_from_fork_nospec_end+0x0/0x39 [ 6882.122705] [] 0xffffffffffffffff [ 6882.123332] LustreError: dumping log to /tmp/lustre-log.1556673683.3720 [ 7476.001071] LNet: Service thread pid 3759 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 7476.001178] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09f61b3050 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:181/0 lens 608/0 e 0 to 0 dl 1556674281 ref 2 fl New:/2/ffffffff rc 0/-1 [ 7476.001182] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 7476.008082] LNet: Skipped 1 previous similar message [ 7476.010086] Pid: 3759, comm: ll_ost_io00_025 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 7476.011625] Call Trace: [ 7476.012138] [] cv_wait_common+0x125/0x150 [spl] [ 7476.013198] [] __cv_wait+0x15/0x20 [spl] [ 7476.014143] [] txg_wait_synced+0xef/0x140 [zfs] [ 7476.015202] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 7476.016235] [] dmu_tx_assign+0x92/0x490 [zfs] [ 7476.017192] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 7476.018266] [] ofd_trans_start+0x75/0xf0 [ofd] [ 7476.019351] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 7476.020400] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 7476.021429] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 7476.022761] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 7476.024004] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 7476.025477] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 7476.026706] [] kthread+0xd1/0xe0 [ 7476.027578] [] ret_from_fork_nospec_end+0x0/0x39 [ 7476.028907] [] 0xffffffffffffffff [ 7476.029923] LustreError: dumping log to /tmp/lustre-log.1556674277.3759 [ 7480.096864] Pid: 3760, comm: ll_ost_io00_026 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 7480.098027] Call Trace: [ 7480.098312] [] cv_wait_common+0x125/0x150 [spl] [ 7480.099018] [] __cv_wait+0x15/0x20 [spl] [ 7480.099627] [] txg_wait_synced+0xef/0x140 [zfs] [ 7480.100419] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 7480.101095] [] dmu_tx_assign+0x92/0x490 [zfs] [ 7480.101773] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 7480.102509] [] ofd_trans_start+0x75/0xf0 [ofd] [ 7480.103179] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 7480.103900] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 7480.104538] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 7480.105299] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 7480.106081] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 7480.106935] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 7480.107637] [] kthread+0xd1/0xe0 [ 7480.108172] [] ret_from_fork_nospec_end+0x0/0x39 [ 7480.108856] [] 0xffffffffffffffff [ 7480.109475] LustreError: dumping log to /tmp/lustre-log.1556674281.3760 [ 7481.027600] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 7481.028800] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 7481.032262] Lustre: 3850:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09f3802050 x1632281923889024/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:181/0 lens 608/0 e 0 to 0 dl 1556674281 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 7481.035061] Lustre: 3850:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 8078.083357] LNet: Service thread pid 3804 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 8078.085168] LNet: Skipped 1 previous similar message [ 8078.085691] Pid: 3804, comm: ll_ost_io00_027 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 8078.086733] Call Trace: [ 8078.087051] [] cv_wait_common+0x125/0x150 [spl] [ 8078.087732] [] __cv_wait+0x15/0x20 [spl] [ 8078.088338] [] txg_wait_synced+0xef/0x140 [zfs] [ 8078.089106] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 8078.089758] [] dmu_tx_assign+0x92/0x490 [zfs] [ 8078.090411] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 8078.091123] [] ofd_trans_start+0x75/0xf0 [ofd] [ 8078.091794] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 8078.092504] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 8078.093145] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 8078.094079] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 8078.094831] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8078.095672] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 8078.096373] [] kthread+0xd1/0xe0 [ 8078.096911] [] ret_from_fork_nospec_end+0x0/0x39 [ 8078.097587] [] 0xffffffffffffffff [ 8078.098244] LustreError: dumping log to /tmp/lustre-log.1556674879.3804 [ 8078.100471] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09f1403850 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:27/0 lens 608/0 e 0 to 0 dl 1556674882 ref 2 fl New:/2/ffffffff rc 0/-1 [ 8078.103739] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 8082.012349] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 8082.013543] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 8082.017238] Lustre: 3890:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09d4d4f050 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:27/0 lens 608/0 e 0 to 0 dl 1556674882 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 8082.020059] Lustre: 3890:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 8082.179099] Pid: 3806, comm: ll_ost_io00_028 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 8082.180179] Call Trace: [ 8082.180447] [] cv_wait_common+0x125/0x150 [spl] [ 8082.181105] [] __cv_wait+0x15/0x20 [spl] [ 8082.181815] [] txg_wait_synced+0xef/0x140 [zfs] [ 8082.182514] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 8082.183216] [] dmu_tx_assign+0x92/0x490 [zfs] [ 8082.183887] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 8082.184580] [] ofd_trans_start+0x75/0xf0 [ofd] [ 8082.185295] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 8082.186012] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 8082.186759] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 8082.187503] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 8082.188294] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8082.189149] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 8082.189851] [] kthread+0xd1/0xe0 [ 8082.190378] [] ret_from_fork_nospec_end+0x0/0x39 [ 8082.191097] [] 0xffffffffffffffff [ 8082.191657] LustreError: dumping log to /tmp/lustre-log.1556674883.3806 [ 8678.116373] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09f1403050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:628/0 lens 608/0 e 0 to 0 dl 1556675483 ref 2 fl New:/2/ffffffff rc 0/-1 [ 8678.119363] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 8680.163981] LNet: Service thread pid 3847 was inactive for 1204.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 8680.165749] LNet: Skipped 1 previous similar message [ 8680.166292] Pid: 3847, comm: ll_ost_io00_029 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 8680.167354] Call Trace: [ 8680.167673] [] cv_wait_common+0x125/0x150 [spl] [ 8680.168370] [] __cv_wait+0x15/0x20 [spl] [ 8680.169009] [] txg_wait_synced+0xef/0x140 [zfs] [ 8680.170007] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 8680.170681] [] dmu_tx_assign+0x92/0x490 [zfs] [ 8680.171361] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 8680.172158] [] ofd_trans_start+0x75/0xf0 [ofd] [ 8680.172837] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 8680.173562] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 8680.174228] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 8680.175166] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 8680.175930] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8680.176785] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 8680.177501] [] kthread+0xd1/0xe0 [ 8680.178067] [] ret_from_fork_nospec_end+0x0/0x39 [ 8680.178746] [] 0xffffffffffffffff [ 8680.179413] LustreError: dumping log to /tmp/lustre-log.1556675481.3847 [ 8682.986373] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 8682.987618] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 8682.990950] Lustre: 3932:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09f1402850 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:628/0 lens 608/0 e 0 to 0 dl 1556675483 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 8682.993896] Lustre: 3932:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 8684.259793] Pid: 3850, comm: ll_ost_io00_030 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 8684.260956] Call Trace: [ 8684.261233] [] cv_wait_common+0x125/0x150 [spl] [ 8684.261920] [] __cv_wait+0x15/0x20 [spl] [ 8684.262521] [] txg_wait_synced+0xef/0x140 [zfs] [ 8684.263237] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 8684.263912] [] dmu_tx_assign+0x92/0x490 [zfs] [ 8684.264575] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 8684.265306] [] ofd_trans_start+0x75/0xf0 [ofd] [ 8684.265979] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 8684.266687] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 8684.267351] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 8684.268114] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 8684.268894] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 8684.269732] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 8684.270451] [] kthread+0xd1/0xe0 [ 8684.271000] [] ret_from_fork_nospec_end+0x0/0x39 [ 8684.271669] [] 0xffffffffffffffff [ 8684.272241] LustreError: dumping log to /tmp/lustre-log.1556675486.3850 [ 9136.896588] serial8250: too much work for irq4 [ 9136.944676] serial8250: too much work for irq4 [ 9136.992691] serial8250: too much work for irq4 [ 9137.040729] serial8250: too much work for irq4 [ 9137.088677] serial8250: too much work for irq4 [ 9137.136707] serial8250: too much work for irq4 [ 9137.184731] serial8250: too much work for irq4 [ 9137.232773] serial8250: too much work for irq4 [ 9137.280770] serial8250: too much work for irq4 [ 9137.328751] serial8250: too much work for irq4 [ 9278.149043] LNet: Service thread pid 3888 was inactive for 1200.10s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 9278.149093] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8c09ea686050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:474/0 lens 608/0 e 0 to 0 dl 1556676084 ref 2 fl New:/2/ffffffff rc 0/-1 [ 9278.149098] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 9278.156096] LNet: Skipped 1 previous similar message [ 9278.158211] Pid: 3888, comm: ll_ost_io00_031 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 9278.159863] Call Trace: [ 9278.160503] [] cv_wait_common+0x125/0x150 [spl] [ 9278.161574] [] __cv_wait+0x15/0x20 [spl] [ 9278.162688] [] txg_wait_synced+0xef/0x140 [zfs] [ 9278.163937] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 9278.165098] [] dmu_tx_assign+0x92/0x490 [zfs] [ 9278.166142] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 9278.167296] [] ofd_trans_start+0x75/0xf0 [ofd] [ 9278.168238] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 9278.169644] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 9278.170786] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 9278.172316] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 9278.173490] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 9278.175150] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 9278.176144] [] kthread+0xd1/0xe0 [ 9278.177219] [] ret_from_fork_nospec_end+0x0/0x39 [ 9278.178649] [] 0xffffffffffffffff [ 9278.179832] LustreError: dumping log to /tmp/lustre-log.1556676079.3888 [ 9283.965518] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 9283.966936] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 9283.970682] Lustre: 3992:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09ea7f5850 x1632281923888736/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:474/0 lens 608/0 e 0 to 0 dl 1556676084 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 9283.973602] Lustre: 3992:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [ 9286.340555] Pid: 3890, comm: ll_ost_io00_032 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 9286.341703] Call Trace: [ 9286.341978] [] cv_wait_common+0x125/0x150 [spl] [ 9286.342665] [] __cv_wait+0x15/0x20 [spl] [ 9286.343391] [] txg_wait_synced+0xef/0x140 [zfs] [ 9286.344178] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 9286.344863] [] dmu_tx_assign+0x92/0x490 [zfs] [ 9286.345598] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 9286.346357] [] ofd_trans_start+0x75/0xf0 [ofd] [ 9286.347094] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 9286.347827] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 9286.348543] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 9286.349338] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 9286.350222] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 9286.351084] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 9286.351833] [] kthread+0xd1/0xe0 [ 9286.352413] [] ret_from_fork_nospec_end+0x0/0x39 [ 9286.353174] [] 0xffffffffffffffff [ 9286.353753] LustreError: dumping log to /tmp/lustre-log.1556676088.3890 [ 9880.230306] LNet: Service thread pid 3930 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 9880.232185] LNet: Skipped 1 previous similar message [ 9880.232739] Pid: 3930, comm: ll_ost_io00_033 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 9880.233821] Call Trace: [ 9880.236510] [] cv_wait_common+0x125/0x150 [spl] [ 9880.237263] [] __cv_wait+0x15/0x20 [spl] [ 9880.237888] [] txg_wait_synced+0xef/0x140 [zfs] [ 9880.238695] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 9880.239469] [] dmu_tx_assign+0x92/0x490 [zfs] [ 9880.240134] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 9880.240986] [] ofd_trans_start+0x75/0xf0 [ofd] [ 9880.241697] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 9880.242482] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 9880.243232] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 9880.244234] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 9880.245086] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 9880.245998] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 9880.246727] [] kthread+0xd1/0xe0 [ 9880.247344] [] ret_from_fork_nospec_end+0x0/0x39 [ 9880.248080] [] 0xffffffffffffffff [ 9880.248830] LustreError: dumping log to /tmp/lustre-log.1556676682.3930 [ 9880.249604] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09d5427050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:320/0 lens 608/0 e 0 to 0 dl 1556676685 ref 2 fl New:/2/ffffffff rc 0/-1 [ 9880.252634] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [ 9884.326056] Pid: 3932, comm: ll_ost_io00_034 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [ 9884.327190] Call Trace: [ 9884.327469] [] cv_wait_common+0x125/0x150 [spl] [ 9884.328144] [] __cv_wait+0x15/0x20 [spl] [ 9884.328821] [] txg_wait_synced+0xef/0x140 [zfs] [ 9884.329538] [] dmu_tx_wait+0x275/0x3c0 [zfs] [ 9884.330276] [] dmu_tx_assign+0x92/0x490 [zfs] [ 9884.330983] [] osd_trans_start+0x199/0x440 [osd_zfs] [ 9884.331772] [] ofd_trans_start+0x75/0xf0 [ofd] [ 9884.332450] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [ 9884.333266] [] ofd_commitrw+0x48c/0x9e0 [ofd] [ 9884.333954] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [ 9884.334769] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [ 9884.335551] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [ 9884.336434] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [ 9884.337174] [] kthread+0xd1/0xe0 [ 9884.337752] [] ret_from_fork_nospec_end+0x0/0x39 [ 9884.338437] [] 0xffffffffffffffff [ 9884.338984] LustreError: dumping log to /tmp/lustre-log.1556676686.3932 [ 9884.937935] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [ 9884.939124] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [ 9884.941532] Lustre: 4066:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c0a37f1a050 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:320/0 lens 608/0 e 0 to 0 dl 1556676685 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [ 9884.944612] Lustre: 4066:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [10480.264002] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09d9faf850 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:166/0 lens 608/0 e 0 to 0 dl 1556677286 ref 2 fl New:/2/ffffffff rc 0/-1 [10480.267301] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [10482.311637] LNet: Service thread pid 3989 was inactive for 1204.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [10482.313678] LNet: Skipped 1 previous similar message [10482.314213] Pid: 3989, comm: ll_ost_io00_035 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [10482.315296] Call Trace: [10482.315653] [] cv_wait_common+0x125/0x150 [spl] [10482.316361] [] __cv_wait+0x15/0x20 [spl] [10482.316996] [] txg_wait_synced+0xef/0x140 [zfs] [10482.317839] [] dmu_tx_wait+0x275/0x3c0 [zfs] [10482.318531] [] dmu_tx_assign+0x92/0x490 [zfs] [10482.319225] [] osd_trans_start+0x199/0x440 [osd_zfs] [10482.319991] [] ofd_trans_start+0x75/0xf0 [ofd] [10482.320731] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [10482.321467] [] ofd_commitrw+0x48c/0x9e0 [ofd] [10482.322149] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [10482.323123] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [10482.323925] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [10482.324803] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [10482.325549] [] kthread+0xd1/0xe0 [10482.326133] [] ret_from_fork_nospec_end+0x0/0x39 [10482.326851] [] 0xffffffffffffffff [10482.327527] LustreError: dumping log to /tmp/lustre-log.1556677284.3989 [10485.915785] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [10485.917016] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [10485.921523] Lustre: 4108:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09d5425050 x1632281923888432/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:166/0 lens 608/0 e 0 to 0 dl 1556677286 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [10485.924529] Lustre: 4108:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [10486.407419] Pid: 3992, comm: ll_ost_io00_036 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [10486.408775] Call Trace: [10486.409058] [] cv_wait_common+0x125/0x150 [spl] [10486.409768] [] __cv_wait+0x15/0x20 [spl] [10486.410414] [] txg_wait_synced+0xef/0x140 [zfs] [10486.411282] [] dmu_tx_wait+0x275/0x3c0 [zfs] [10486.412046] [] dmu_tx_assign+0x92/0x490 [zfs] [10486.412751] [] osd_trans_start+0x199/0x440 [osd_zfs] [10486.413551] [] ofd_trans_start+0x75/0xf0 [ofd] [10486.414265] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [10486.415053] [] ofd_commitrw+0x48c/0x9e0 [ofd] [10486.415746] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [10486.416561] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [10486.417400] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [10486.418355] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [10486.419130] [] kthread+0xd1/0xe0 [10486.419700] [] ret_from_fork_nospec_end+0x0/0x39 [10486.420429] [] 0xffffffffffffffff [10486.421027] LustreError: dumping log to /tmp/lustre-log.1556677288.3992 [11080.296846] Pid: 4062, comm: ll_ost_io00_037 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [11080.300346] Call Trace: [11080.300682] [] cv_wait_common+0x125/0x150 [spl] [11080.301396] [] __cv_wait+0x15/0x20 [spl] [11080.302111] [] txg_wait_synced+0xef/0x140 [zfs] [11080.302932] [] dmu_tx_wait+0x275/0x3c0 [zfs] [11080.303606] [] dmu_tx_assign+0x92/0x490 [zfs] [11080.304335] [] osd_trans_start+0x199/0x440 [osd_zfs] [11080.305081] [] ofd_trans_start+0x75/0xf0 [ofd] [11080.305883] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [11080.306598] [] ofd_commitrw+0x48c/0x9e0 [ofd] [11080.307257] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [11080.308222] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [11080.309006] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [11080.309862] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [11080.310568] [] kthread+0xd1/0xe0 [11080.311129] [] ret_from_fork_nospec_end+0x0/0x39 [11080.311828] [] 0xffffffffffffffff [11080.312499] LustreError: dumping log to /tmp/lustre-log.1556677882.4062 [11082.344818] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09dd3e5050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:12/0 lens 608/0 e 0 to 0 dl 1556677887 ref 2 fl New:/2/ffffffff rc 0/-1 [11082.349477] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [11086.891910] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [11086.893129] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [11086.896548] Lustre: 4150:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09db281850 x1632281923888880/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:12/0 lens 608/0 e 0 to 0 dl 1556677887 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [11086.899448] Lustre: 4150:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 5 previous similar messages [11088.488388] LNet: Service thread pid 4067 was inactive for 1203.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [11088.490183] LNet: Skipped 2 previous similar messages [11088.490742] Pid: 4067, comm: ll_ost_io00_039 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [11088.491950] Call Trace: [11088.492236] [] cv_wait_common+0x125/0x150 [spl] [11088.492979] [] __cv_wait+0x15/0x20 [spl] [11088.493598] [] txg_wait_synced+0xef/0x140 [zfs] [11088.494358] [] dmu_tx_wait+0x275/0x3c0 [zfs] [11088.495017] [] dmu_tx_assign+0x92/0x490 [zfs] [11088.495684] [] osd_trans_start+0x199/0x440 [osd_zfs] [11088.496407] [] ofd_trans_start+0x75/0xf0 [ofd] [11088.497065] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [11088.497780] [] ofd_commitrw+0x48c/0x9e0 [ofd] [11088.498445] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [11088.499196] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [11088.499992] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [11088.500844] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [11088.501559] [] kthread+0xd1/0xe0 [11088.502095] [] ret_from_fork_nospec_end+0x0/0x39 [11088.502773] [] 0xffffffffffffffff [11088.503354] LustreError: dumping log to /tmp/lustre-log.1556677890.4067 [11088.646872] Pid: 4066, comm: ll_ost_io00_038 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [11088.647980] Call Trace: [11088.648255] [] cv_wait_common+0x125/0x150 [spl] [11088.648938] [] __cv_wait+0x15/0x20 [spl] [11088.649558] [] txg_wait_synced+0xef/0x140 [zfs] [11088.650252] [] dmu_tx_wait+0x275/0x3c0 [zfs] [11088.650923] [] dmu_tx_assign+0x92/0x490 [zfs] [11088.651598] [] osd_trans_start+0x199/0x440 [osd_zfs] [11088.652499] [] ofd_trans_start+0x75/0xf0 [ofd] [11088.653214] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [11088.654005] [] ofd_commitrw+0x48c/0x9e0 [ofd] [11088.654668] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [11088.655470] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [11088.656321] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [11088.657239] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [11088.658007] [] kthread+0xd1/0xe0 [11088.658557] [] ret_from_fork_nospec_end+0x0/0x39 [11088.659292] [] 0xffffffffffffffff [11682.378127] Pid: 4105, comm: ll_ost_io00_040 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [11682.378243] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09da715050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:613/0 lens 608/0 e 0 to 0 dl 1556678488 ref 2 fl New:/2/ffffffff rc 0/-1 [11682.378248] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [11682.383245] Call Trace: [11682.383565] [] cv_wait_common+0x125/0x150 [spl] [11682.384293] [] __cv_wait+0x15/0x20 [spl] [11682.384903] [] txg_wait_synced+0xef/0x140 [zfs] [11682.385809] [] dmu_tx_wait+0x275/0x3c0 [zfs] [11682.388851] [] dmu_tx_assign+0x92/0x490 [zfs] [11682.389527] [] osd_trans_start+0x199/0x440 [osd_zfs] [11682.390282] [] ofd_trans_start+0x75/0xf0 [ofd] [11682.390973] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [11682.391750] [] ofd_commitrw+0x48c/0x9e0 [ofd] [11682.392500] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [11682.393543] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [11682.394379] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [11682.395264] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [11682.396001] [] kthread+0xd1/0xe0 [11682.396632] [] ret_from_fork_nospec_end+0x0/0x39 [11682.397326] [] 0xffffffffffffffff [11682.398040] LustreError: dumping log to /tmp/lustre-log.1556678484.4105 [11686.473875] Pid: 4108, comm: ll_ost_io00_041 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [11686.475001] Call Trace: [11686.475285] [] cv_wait_common+0x125/0x150 [spl] [11686.475965] [] __cv_wait+0x15/0x20 [spl] [11686.476571] [] txg_wait_synced+0xef/0x140 [zfs] [11686.477272] [] dmu_tx_wait+0x275/0x3c0 [zfs] [11686.477938] [] dmu_tx_assign+0x92/0x490 [zfs] [11686.478598] [] osd_trans_start+0x199/0x440 [osd_zfs] [11686.479325] [] ofd_trans_start+0x75/0xf0 [ofd] [11686.479997] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [11686.480702] [] ofd_commitrw+0x48c/0x9e0 [ofd] [11686.481351] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [11686.482098] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [11686.482892] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [11686.483725] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [11686.484441] [] kthread+0xd1/0xe0 [11686.484976] [] ret_from_fork_nospec_end+0x0/0x39 [11686.485643] [] 0xffffffffffffffff [11686.486195] LustreError: dumping log to /tmp/lustre-log.1556678488.4108 [11687.865741] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [11687.866944] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [11687.870340] Lustre: 4193:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09da714050 x1632281923888512/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:613/0 lens 608/0 e 0 to 0 dl 1556678488 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [11687.873237] Lustre: 4193:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [12284.459761] LNet: Service thread pid 4149 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [12284.461868] LNet: Skipped 3 previous similar messages [12284.462425] Pid: 4149, comm: ll_ost_io00_042 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [12284.463541] Call Trace: [12284.463885] [] cv_wait_common+0x125/0x150 [spl] [12284.464616] [] __cv_wait+0x15/0x20 [spl] [12284.465514] [] txg_wait_synced+0xef/0x140 [zfs] [12284.466391] [] dmu_tx_wait+0x275/0x3c0 [zfs] [12284.467125] [] dmu_tx_assign+0x92/0x490 [zfs] [12284.467881] [] osd_trans_start+0x199/0x440 [osd_zfs] [12284.468692] [] ofd_trans_start+0x75/0xf0 [ofd] [12284.469503] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [12284.470264] [] ofd_commitrw+0x48c/0x9e0 [ofd] [12284.471023] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [12284.472077] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [12284.472959] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [12284.473903] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [12284.474663] [] kthread+0xd1/0xe0 [12284.475346] [] ret_from_fork_nospec_end+0x0/0x39 [12284.476073] [] 0xffffffffffffffff [12284.476812] LustreError: dumping log to /tmp/lustre-log.1556679086.4149 [12284.479506] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09f61b3050 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:459/0 lens 608/0 e 0 to 0 dl 1556679089 ref 2 fl New:/2/ffffffff rc 0/-1 [12284.484214] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [12288.555533] Pid: 4150, comm: ll_ost_io00_043 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [12288.556694] Call Trace: [12288.556978] [] cv_wait_common+0x125/0x150 [spl] [12288.557650] [] __cv_wait+0x15/0x20 [spl] [12288.558255] [] txg_wait_synced+0xef/0x140 [zfs] [12288.558959] [] dmu_tx_wait+0x275/0x3c0 [zfs] [12288.559625] [] dmu_tx_assign+0x92/0x490 [zfs] [12288.560302] [] osd_trans_start+0x199/0x440 [osd_zfs] [12288.561027] [] ofd_trans_start+0x75/0xf0 [ofd] [12288.561699] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [12288.562704] [] ofd_commitrw+0x48c/0x9e0 [ofd] [12288.563382] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [12288.564211] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [12288.564991] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [12288.565859] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [12288.566628] [] kthread+0xd1/0xe0 [12288.567189] [] ret_from_fork_nospec_end+0x0/0x39 [12288.567868] [] 0xffffffffffffffff [12288.568470] LustreError: dumping log to /tmp/lustre-log.1556679090.4150 [12288.838947] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [12288.840162] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [12288.843409] Lustre: 4235:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09f3802050 x1632281923888512/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:459/0 lens 608/0 e 0 to 0 dl 1556679089 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [12288.846324] Lustre: 4235:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [12882.445879] Pid: 4190, comm: ll_ost_io00_044 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [12882.447103] Call Trace: [12882.447427] [] cv_wait_common+0x125/0x150 [spl] [12882.448135] [] __cv_wait+0x15/0x20 [spl] [12882.448733] [] txg_wait_synced+0xef/0x140 [zfs] [12882.449529] [] dmu_tx_wait+0x275/0x3c0 [zfs] [12882.450206] [] dmu_tx_assign+0x92/0x490 [zfs] [12882.450871] [] osd_trans_start+0x199/0x440 [osd_zfs] [12882.451603] [] ofd_trans_start+0x75/0xf0 [ofd] [12882.452294] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [12882.453008] [] ofd_commitrw+0x48c/0x9e0 [ofd] [12882.453656] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [12882.454610] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [12882.455394] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [12882.456245] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [12882.456951] [] kthread+0xd1/0xe0 [12882.457504] [] ret_from_fork_nospec_end+0x0/0x39 [12882.458195] [] 0xffffffffffffffff [12882.458872] LustreError: dumping log to /tmp/lustre-log.1556679684.4190 [12884.493840] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09f1407050 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:305/0 lens 608/0 e 0 to 0 dl 1556679690 ref 2 fl New:/2/ffffffff rc 0/-1 [12884.496792] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [12889.813911] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [12889.815113] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [12889.818704] Lustre: 4289:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09d4d49050 x1632281923889024/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:305/0 lens 608/0 e 0 to 0 dl 1556679690 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [12889.821618] Lustre: 4289:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [12890.637371] LNet: Service thread pid 4193 was inactive for 1202.82s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [12890.639178] LNet: Skipped 2 previous similar messages [12890.639728] Pid: 4193, comm: ll_ost_io00_045 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [12890.640909] Call Trace: [12890.641186] [] cv_wait_common+0x125/0x150 [spl] [12890.641874] [] __cv_wait+0x15/0x20 [spl] [12890.642550] [] txg_wait_synced+0xef/0x140 [zfs] [12890.643257] [] dmu_tx_wait+0x275/0x3c0 [zfs] [12890.643933] [] dmu_tx_assign+0x92/0x490 [zfs] [12890.644612] [] osd_trans_start+0x199/0x440 [osd_zfs] [12890.645325] [] ofd_trans_start+0x75/0xf0 [ofd] [12890.646001] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [12890.646723] [] ofd_commitrw+0x48c/0x9e0 [ofd] [12890.647380] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [12890.648134] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [12890.648912] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [12890.649764] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [12890.650470] [] kthread+0xd1/0xe0 [12890.651003] [] ret_from_fork_nospec_end+0x0/0x39 [12890.651677] [] 0xffffffffffffffff [12890.652229] LustreError: dumping log to /tmp/lustre-log.1556679692.4193 [13486.575881] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09ea7f3050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:151/0 lens 608/0 e 0 to 0 dl 1556680291 ref 2 fl New:/2/ffffffff rc 0/-1 [13486.580030] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [13488.623570] Pid: 4232, comm: ll_ost_io00_046 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [13488.624722] Call Trace: [13488.625048] [] cv_wait_common+0x125/0x150 [spl] [13488.625757] [] __cv_wait+0x15/0x20 [spl] [13488.626362] [] txg_wait_synced+0xef/0x140 [zfs] [13488.627159] [] dmu_tx_wait+0x275/0x3c0 [zfs] [13488.627833] [] dmu_tx_assign+0x92/0x490 [zfs] [13488.628512] [] osd_trans_start+0x199/0x440 [osd_zfs] [13488.629261] [] ofd_trans_start+0x75/0xf0 [ofd] [13488.629954] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [13488.630680] [] ofd_commitrw+0x48c/0x9e0 [ofd] [13488.631332] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [13488.632270] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [13488.633057] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [13488.633915] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [13488.634640] [] kthread+0xd1/0xe0 [13488.635198] [] ret_from_fork_nospec_end+0x0/0x39 [13488.635885] [] 0xffffffffffffffff [13488.636551] LustreError: dumping log to /tmp/lustre-log.1556680290.4232 [13490.789660] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [13490.791162] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [13490.794861] Lustre: 4330:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09f1404050 x1632281923888512/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:151/0 lens 608/0 e 0 to 0 dl 1556680291 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [13490.797769] Lustre: 4330:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [13492.719345] LNet: Service thread pid 4235 was inactive for 1203.93s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [13492.721394] LNet: Skipped 1 previous similar message [13492.721920] Pid: 4235, comm: ll_ost_io00_047 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [13492.723035] Call Trace: [13492.723386] [] cv_wait_common+0x125/0x150 [spl] [13492.724246] [] __cv_wait+0x15/0x20 [spl] [13492.724952] [] txg_wait_synced+0xef/0x140 [zfs] [13492.725713] [] dmu_tx_wait+0x275/0x3c0 [zfs] [13492.726467] [] dmu_tx_assign+0x92/0x490 [zfs] [13492.727216] [] osd_trans_start+0x199/0x440 [osd_zfs] [13492.728050] [] ofd_trans_start+0x75/0xf0 [ofd] [13492.728799] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [13492.729603] [] ofd_commitrw+0x48c/0x9e0 [ofd] [13492.730351] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [13492.731371] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [13492.732174] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [13492.733078] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [13492.733824] [] kthread+0xd1/0xe0 [13492.734412] [] ret_from_fork_nospec_end+0x0/0x39 [13492.735129] [] 0xffffffffffffffff [13492.735722] LustreError: dumping log to /tmp/lustre-log.1556680294.4235 [14086.609810] Pid: 4288, comm: ll_ost_io00_048 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [14086.611025] Call Trace: [14086.611355] [] cv_wait_common+0x125/0x150 [spl] [14086.612060] [] __cv_wait+0x15/0x20 [spl] [14086.612668] [] txg_wait_synced+0xef/0x140 [zfs] [14086.615532] [] dmu_tx_wait+0x275/0x3c0 [zfs] [14086.616338] [] dmu_tx_assign+0x92/0x490 [zfs] [14086.617196] [] osd_trans_start+0x199/0x440 [osd_zfs] [14086.617996] [] ofd_trans_start+0x75/0xf0 [ofd] [14086.618739] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [14086.619463] [] ofd_commitrw+0x48c/0x9e0 [ofd] [14086.620131] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [14086.621101] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [14086.621896] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [14086.622761] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [14086.623476] [] kthread+0xd1/0xe0 [14086.624043] [] ret_from_fork_nospec_end+0x0/0x39 [14086.624749] [] 0xffffffffffffffff [14086.625420] LustreError: dumping log to /tmp/lustre-log.1556680888.4288 [14086.627132] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8c09d5421850 x1632281923888592/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:752/0 lens 608/0 e 0 to 0 dl 1556680892 ref 2 fl New:/2/ffffffff rc 0/-1 [14086.630428] Lustre: 19789:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [14090.705540] Pid: 4289, comm: ll_ost_io00_049 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [14090.706711] Call Trace: [14090.706995] [] cv_wait_common+0x125/0x150 [spl] [14090.707673] [] __cv_wait+0x15/0x20 [spl] [14090.708353] [] txg_wait_synced+0xef/0x140 [zfs] [14090.709077] [] dmu_tx_wait+0x275/0x3c0 [zfs] [14090.709858] [] dmu_tx_assign+0x92/0x490 [zfs] [14090.710588] [] osd_trans_start+0x199/0x440 [osd_zfs] [14090.711352] [] ofd_trans_start+0x75/0xf0 [ofd] [14090.712085] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [14090.712803] [] ofd_commitrw+0x48c/0x9e0 [ofd] [14090.713501] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [14090.714317] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [14090.715143] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [14090.716000] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [14090.716763] [] kthread+0xd1/0xe0 [14090.717386] [] ret_from_fork_nospec_end+0x0/0x39 [14090.718134] [] 0xffffffffffffffff [14090.718694] LustreError: dumping log to /tmp/lustre-log.1556680892.4289 [14091.765755] Lustre: lustre-OST0000: Client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) reconnecting [14091.766963] Lustre: lustre-OST0000: Connection restored to 5cf3fa52-eafd-d3e9-da55-a6790393eb07 (at 10.9.6.156@tcp) [14091.770444] Lustre: 4374:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:1s); client may timeout. req@ffff8c09ea686050 x1632281923888512/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:752/0 lens 608/0 e 0 to 0 dl 1556680892 ref 1 fl Interpret:/2/ffffffff rc 0/-1 [14091.773361] Lustre: 4374:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 6 previous similar messages [14548.426623] Lustre: lustre-OST0001: haven't heard from client 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp) in 232 seconds. I think it's dead, and I am evicting it. exp ffff8c0a3a4b2800, cur 1556681350 expire 1556681200 last 1556681118 [14667.396257] serial8250_interrupt: 7 callbacks suppressed [14667.396848] serial8250: too much work for irq4 [14667.472113] serial8250: too much work for irq4 [14667.519989] serial8250: too much work for irq4 [14667.567930] serial8250: too much work for irq4 [14688.691817] LNet: Service thread pid 4328 was inactive for 1202.17s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [14688.693580] LNet: Skipped 2 previous similar messages [14688.694130] Pid: 4328, comm: ll_ost_io00_050 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [14688.695318] Call Trace: [14688.695688] [] cv_wait_common+0x125/0x150 [spl] [14688.696459] [] __cv_wait+0x15/0x20 [spl] [14688.697080] [] txg_wait_synced+0xef/0x140 [zfs] [14688.697933] [] dmu_tx_wait+0x275/0x3c0 [zfs] [14688.698655] [] dmu_tx_assign+0x92/0x490 [zfs] [14688.699408] [] osd_trans_start+0x199/0x440 [osd_zfs] [14688.700159] [] ofd_trans_start+0x75/0xf0 [ofd] [14688.701033] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [14688.701816] [] ofd_commitrw+0x48c/0x9e0 [ofd] [14688.702531] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [14688.703547] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [14688.704387] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [14688.705278] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [14688.706052] [] kthread+0xd1/0xe0 [14688.706664] [] ret_from_fork_nospec_end+0x0/0x39 [14688.707429] [] 0xffffffffffffffff [14688.708114] LustreError: dumping log to /tmp/lustre-log.1556681490.4328 [14688.733011] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8c09dd3e3050 x1632281923888320/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:598/0 lens 608/0 e 0 to 0 dl 1556681493 ref 2 fl New:/2/ffffffff rc 0/-1 [14688.735916] Lustre: 12761:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages [14688.739011] LustreError: 4492:0:(events.c:450:server_bulk_callback()) event type 5, status -111, desc ffff8c09d79e5c00 [14691.737670] LustreError: 4430:0:(ldlm_lib.c:3243:target_bulk_io()) @@@ timeout on bulk WRITE after 3+0s req@ffff8c09d5425050 x1632281923888960/t0(0) o4->9c459574-c33a-63ee-b0e7-6869647e7848@10.9.6.156@tcp:598/0 lens 608/448 e 0 to 0 dl 1556681493 ref 1 fl Interpret:/2/0 rc 0/0 [14691.740241] Lustre: lustre-OST0000: Bulk IO write error with 9c459574-c33a-63ee-b0e7-6869647e7848 (at 10.9.6.156@tcp), client will retry: rc = -110 [14691.742168] LustreError: 4495:0:(events.c:450:server_bulk_callback()) event type 5, status -111, desc ffff8c09d79e5c00 [14691.743406] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d79e5c00 [14691.744779] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d79e5c00 [14691.747795] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d5b27a00 [14691.750281] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d5b27a00 [14691.751990] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d5b27a00 [14691.753989] LustreError: 4430:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8c09d5b27a00 [14692.787595] Pid: 4330, comm: ll_ost_io00_051 3.10.0-957.10.1.el7_lustre.x86_64 #1 SMP Mon Apr 22 22:25:47 UTC 2019 [14692.788695] Call Trace: [14692.788972] [] cv_wait_common+0x125/0x150 [spl] [14692.789669] [] __cv_wait+0x15/0x20 [spl] [14692.790274] [] txg_wait_synced+0xef/0x140 [zfs] [14692.790992] [] dmu_tx_wait+0x275/0x3c0 [zfs] [14692.791664] [] dmu_tx_assign+0x92/0x490 [zfs] [14692.792335] [] osd_trans_start+0x199/0x440 [osd_zfs] [14692.793056] [] ofd_trans_start+0x75/0xf0 [ofd] [14692.793720] [] ofd_commitrw_write+0xa31/0x1d40 [ofd] [14692.794429] [] ofd_commitrw+0x48c/0x9e0 [ofd] [14692.795087] [] tgt_brw_write+0x10cc/0x1cf0 [ptlrpc] [14692.795835] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [14692.796613] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [14692.797456] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [14692.798166] [] kthread+0xd1/0xe0 [14692.798703] [] ret_from_fork_nospec_end+0x0/0x39 [14692.799372] [] 0xffffffffffffffff [14692.799930] LustreError: dumping log to /tmp/lustre-log.1556681494.4330 [14870.328211] serial8250: too much work for irq4 [14870.404124] serial8250: too much work for irq4 [14870.503078] serial8250: too much work for irq4